Question Downloadable Japanese vocabulary lists?

As a hobby, I'm currently building a japanese vocabulary learning tool.
But for that I need a vocabulary list. The information that i would like to include in my list are:
Kanji | Reading (Hira) | Romaji | Translation | Word Type | Word Category | Frequency | Level (e.g JLPT)
"Word Type" should be like noun, adjective, verb, etc...
The "category" should be a thematic categorization of the word, like "biology", "chemistry" or "movement" and "communication" for verbs. for offering specific lections.

I've already spent some days to do research but i didn't find a good solution. here is what i already tried:
1. JMDict. But sadly JMDict is not a vocabulary list but a dictionary. a lot of translations are more like "sentences" and the schema is incredibly unhelpful. every enty can list multiple kanji, multiple readings and multiple translations, without a clear connection between them, you don'tknow which reading corresponds to what kanji and what translation.
2. BCCWJ. this is a list that includes frequency data very detailed
3. wadoku XML dump
4. various online JLPT sources.

Now what is my biggest Problem? the join!
If you know the japanese language, you'll probably know: The combination of Kanji + Reading is not unique
so let's just take the letter A. apparently the letter A is used for various onomatopoeia, but is also the word for certain diseases. what now happens is that If i try to join and of the data sources, that i can't really distinct between them, because there's no clear identification of the "sense" of the word. so e.g. the word "A" in the meaning of "Ah!" as an expression of pain, is mapped to "pharmacology"

the same problem would then obviously exist with frequency data and JLPT levels. The best way i currently worked out is... Ask ChatGPT
but it'd take days, if not months if i were to use AI tools for a 100k data set like JMDict. and it'd also take eons to generate such a data set from scratch, especially on free tiers.

So now i wonder what's your oppinion of this topic, do you know similar projects out there, did you work with JMDict before and have a better understanding of the structure? any help is appreciated!

Check out Quizlet. It might already cover most of what you're trying to do.

mdchachi said:

Check out Quizlet. It might already cover most of what you're trying to do.

https://quizlet.com/subjects/languages/japanese-flashcards-decbd26c-t01

Click to expand...

and can you download stuff with it?

Kokujou,

When I was teaching in Japan, I did pretty much the same thing that you are doing now. (But I used my topic-centered vocabulary lists for English speaking activities, not for creating an app. (I assume that by "vocabulary leaning tool" you mean an app.)

Your first step is to make a list of all the topics (chemistry, biology, etc.) that you wish to create vocabulary lists for. Have you already done this? How many topics have you decided to work on? Please show us the entire list of topics you have chosen to work on.

Buntaro said:

Kokujou,

When I was teaching in Japan, I did pretty much the same thing that you are doing now. (But I used my topic-centered vocabulary lists for English speaking activities, not for creating an app. (I assume that by "vocabulary leaning tool" you mean an app.)

Your first step is to make a list of all the topics (chemistry, biology, etc.) that you wish to create vocabulary lists for. Have you already done this? How many topics have you decided to work on? Please show us the entire list of topics you have chosen to work on.

Click to expand...

i didn't really decide on concrete topics yet, i worked up a bit up a list using ChatGPT for ideas as well as the list that the JMDict includes, but i'm not sure if i have enough coverage or too much coverage, so please consider this a temporary draft

Undefined, Animals, Traffic, Places, Clothing, Colors, People, Professions, Society, Art, Drinks, Food, Home, Electronics, Body, Nature, Materials, Mathematics, Directions, Seasons, Numbers, Feelings, Time, Physics, Movement, Communication, Technology, Biology, Chemistry, Sports, Hobbies, Games, Psychology, SocialStudies, Geography, Geology, Anatomy, Perception, Skills, Philosophy, Jurisprudence, Ethics, Sounds, Basics, Religion, Music, Cooking, Astronomy, Finance, Architecture, Mythology, Geometry, Agriculture

ideally i would've liked to have a relatively complete database that i can then reduce by e.g. filtering out word types that are more grammar based, or that are too uncommon or too specialized or so.

in case you're interested, i'm writing a web application with frontend and backend. but since it involves backend it's likely it won't be publicly available, except the github repostiory ^^

johnnyG said:

How about here?

Click to expand...

the problem is not the single property. i have the BCCWJ list for frequencies, which is really big and most likely sufficient, but: how to join them?
i need all the properties in one list, and if the combination of kanji and reading is not the same, there is no possible way to know to which sense those entries correspond.

this is my biggest problem, i have the frequencies from BCCWJ, the JLPT list from different sources and the JMDict for categories and word types. but if i just join them on the first occurrence i get a lot of wrong information.

You can also take shared anki decks and see what you can do with them.
The old 2k/6k decks originally from iknow who opened them many years ago (2k meaning 2k most common words)
JLPT
grammar

As far as I know all the data is fine for you to use in an app you are building.

yeah those i can read, but i'm not sure if they have the schema i need...
they do have translations and kanji etc... they even have the word type, but as far as i see, please correct me, they don't have a thematic categorization or frequency data...

so the problem would be the same as with JMDict, how to join the remaining data in?

the base problem remains :/
but thanks maybe i can do something with them on the long run!