This works really well with stories like 君の名は where the movie is full of visuals and non-dialogue scenes. Anime series may go out of order between what's presented in the books (Goblin Slayer for instance), but that's not too bad as you will remember locations where actions took place and feel the atmosphere as the words come to life in your mind's eye. Some people don't like to do this as it "spoils" the story. However, if it's a good story then it will be good on a reread or reading after you watched the adaption. Still, it's up to you on what you do. Eventually, yes, you need to move to more dense books and even non-fiction. However, for your trek to N1 level reading comprehension, light novels are a great source. Perhaps make a goal of 50 light novels between N4 and N1 levels of learning.