As a hobby, I'm currently building a japanese vocabulary learning tool.
But for that I need a vocabulary list. The information that i would like to include in my list are:
Kanji | Reading (Hira) | Romaji | Translation | Word Type | Word Category | Frequency | Level (e.g JLPT)
"Word Type" should be like noun, adjective, verb, etc...
The "category" should be a thematic categorization of the word, like "biology", "chemistry" or "movement" and "communication" for verbs. for offering specific lections.
I've already spent some days to do research but i didn't find a good solution. here is what i already tried:
1. JMDict. But sadly JMDict is not a vocabulary list but a dictionary. a lot of translations are more like "sentences" and the schema is incredibly unhelpful. every enty can list multiple kanji, multiple readings and multiple translations, without a clear connection between them, you don'tknow which reading corresponds to what kanji and what translation.
2. BCCWJ. this is a list that includes frequency data very detailed
3. wadoku XML dump
4. various online JLPT sources.
Now what is my biggest Problem? the join!
If you know the japanese language, you'll probably know: The combination of Kanji + Reading is not unique
so let's just take the letter A. apparently the letter A is used for various onomatopoeia, but is also the word for certain diseases. what now happens is that If i try to join and of the data sources, that i can't really distinct between them, because there's no clear identification of the "sense" of the word. so e.g. the word "A" in the meaning of "Ah!" as an expression of pain, is mapped to "pharmacology"
the same problem would then obviously exist with frequency data and JLPT levels. The best way i currently worked out is... Ask ChatGPT
but it'd take days, if not months if i were to use AI tools for a 100k data set like JMDict. and it'd also take eons to generate such a data set from scratch, especially on free tiers.
So now i wonder what's your oppinion of this topic, do you know similar projects out there, did you work with JMDict before and have a better understanding of the structure? any help is appreciated!
But for that I need a vocabulary list. The information that i would like to include in my list are:
Kanji | Reading (Hira) | Romaji | Translation | Word Type | Word Category | Frequency | Level (e.g JLPT)
"Word Type" should be like noun, adjective, verb, etc...
The "category" should be a thematic categorization of the word, like "biology", "chemistry" or "movement" and "communication" for verbs. for offering specific lections.
I've already spent some days to do research but i didn't find a good solution. here is what i already tried:
1. JMDict. But sadly JMDict is not a vocabulary list but a dictionary. a lot of translations are more like "sentences" and the schema is incredibly unhelpful. every enty can list multiple kanji, multiple readings and multiple translations, without a clear connection between them, you don'tknow which reading corresponds to what kanji and what translation.
2. BCCWJ. this is a list that includes frequency data very detailed
3. wadoku XML dump
4. various online JLPT sources.
Now what is my biggest Problem? the join!
If you know the japanese language, you'll probably know: The combination of Kanji + Reading is not unique
so let's just take the letter A. apparently the letter A is used for various onomatopoeia, but is also the word for certain diseases. what now happens is that If i try to join and of the data sources, that i can't really distinct between them, because there's no clear identification of the "sense" of the word. so e.g. the word "A" in the meaning of "Ah!" as an expression of pain, is mapped to "pharmacology"
the same problem would then obviously exist with frequency data and JLPT levels. The best way i currently worked out is... Ask ChatGPT
but it'd take days, if not months if i were to use AI tools for a 100k data set like JMDict. and it'd also take eons to generate such a data set from scratch, especially on free tiers.
So now i wonder what's your oppinion of this topic, do you know similar projects out there, did you work with JMDict before and have a better understanding of the structure? any help is appreciated!
