guess the last phonemes of a French word
git clone
Log | Files | Refs | README

commit 43892774f45ccf108f0f0b9409c4db4ebe20ec67
parent 87c740896784f74f58495a0dff3de29e576a1dc2
Author: Antoine Amarilli <>
Date:   Tue, 13 Mar 2012 14:38:01 +0100

continue README

README | 91+++++++++++++++++--------------------------------------------------------------
1 file changed, 19 insertions(+), 72 deletions(-)

diff --git a/README b/README @@ -1,5 +1,5 @@ frhyme -- a toolkit to guess the last phonemes of a French word -Copyright (C) 2011 by Antoine Amarilli +Copyright (C) 2011-2012 by Antoine Amarilli == 0. Licence == @@ -34,81 +34,28 @@ the longest common prefix, using a trie for internal representation. To avoid licensing headaches, and because the data file is quite big, no pronunciation data is included, you have to generate it yourself. See section 3. -Once you have pronunciation data ready in -If you just want to use the included training data, you can either run, giving one word per line in stdin and getting the -annotation on stout, or you can import it in a Python file and call -haspirater.lookup(word) which returns True if the leading 'h' is -aspirated, False if it isn't, and raises ValueError if there is no -leading 'h'. +Once you have pronunciation data ready in frhyme.json, you can either run NBEST, giving one word per line in stdin and getting the NBEST top +pronunciations on stdout, or you can import it in a Python file and call +frhyme.lookup(word, NBEST) which returns the NBEST top pronunciations. -Please report any errors in the training data, keeping in mind than only -one possibility is returned even when both are attested. +The pronunciations returned are annotated with a confidence score (the number of +occurrences in the training data). They should be sensible up to the longest +prefix occuring in the training data, but may be prefixed by garbage. == 3. Training == -The training data used by is loaded at runtime from the -haspirater.json file which has been trained from French texts taken from -Project Gutenberg <>, from the list in the Wikipedia -article <>, from the -categories in the French Wiktionary -<égorie:Mots_à_h_muet> and -<égorie:Mots_à_h_aspiré> and from a custom -set of exceptions. If you want to create your own data, or adapt the -approach here to other linguistic features, read on. +The data used by is loaded at runtime from the haspirater.json file +which should be trained from a pronunciation database. The recommended way to do +so is to use a tweaked Lexique <> along with a provided bugfix +file, as follows: -The master script is which accepts French text on stdin and a -list of exceptions files as arguments. Included exception files are -additions and wikipedia. These exceptions are just like training data -and are not stored as-is; they are just piped later on in the training -phase. produces on stdout the json trie. Thus, you would run -something like: + lexique/ > lexique + ./ NPHON lexique additions > frhyme.json - $ cat corpus | ./ exceptions > haspirater.json - - - - - -== 4. Training details == - -=== 4.1. Corpus preparation ( === - -This script removes useless characters, and separates words (one per -line). - -=== 4.2. Property inference ( === - -This script examines the output, notices occurrences of words for which -the preceding word indicates the aspirated or non-aspirated status, and -outputs them. - -=== 4.3. Removing leading 'h' === - -This is a quick optimization. - -=== 4.4. Trie construction ( === - -The occurrences are read one after the other and are used to populate a -trie carrying the value count for each occurrence having a given prefix. - -=== 4.5. Trie compression ( === - -The trie is then compressed by removing branches which are not needed to -infer a value. This step could be followed by a removal of branches with -very little dissent from the majority value if we wanted to reduce the -trie size at the expense of accuracy: for aspirated h, this isn't -needed. - -=== 4.5. Trie majority relabeling ( === - -Instead of the list of values with their counts, nodes are relabeled to -carry the most common value. This step could be skipped to keep -confidence values. - -== 5. Additionnal stuff == - -You can use to convert the output of or in the dot format which can be used to render a drawing -of the trie. The result of such a drawing is given as aspirated_h.pdf +where NPHON is the number of trailing phonemes to keep. Beware, this may take up +several hundred megabytes of RAM. The resulting file should be accurate on the +French words of Lexique, and will return pronunciations in a variant of X-SAMPA +which ensures that each phoneme is mapped to exactly one ASCII character: the +substitutions are "A~" => "#", "O~" => "$", "E~" => ")", "9~" => "(".