It print 1, 2 and 3 … but still not 4 ! It freezes.
Seems like getGCRF() is working.
Besides, you can notice that I included a getITRF() in my main() function which has a good behaviour. So my orekit-data-master folder contains all data necessary for ITRF.
Sorry to read that. Seems like the problem is quite complex.
Is there a way to properly debug with starmap and see where the code stalls?
(I’m a bit out of idea…)
Are your two Python environments equivalent in terms of library versions (check with pip freeze for example)? If not try aligning the non working one on the other with a requirement file
Very hard to see where the problem could be. I have mostly used threads, but notice there you need to do an AttachCurrentThread() to get the JVM access. Not completely sure why you don’t seem to need that here (try!). See for example: Python wrapper and orekit.initVM()
Not sure if different java versions could have an influence, I typically use openjdk 8 which is what is installed with the default conda orekit.
Are your two Python environments equivalent in terms of library versions (check with pip freeze for example)? If not try aligning the non working one on the other with a requirement file
@Serrof Yes they are. I use python 3.10 and a pip freeze gives the following output in both cases:
orekit==11.3.2
orekit-stubs==0.1
Very hard to see where the problem could be. I have mostly used threads, but notice there you need to do an AttachCurrentThread() to get the JVM access. Not completely sure why you don’t seem to need that here (try!). See for example: Python wrapper and orekit.initVM()
Not sure if different java versions could have an influence, I typically use openjdk 8 which is what is installed with the default conda orekit.
@petrus.hyvonen I can’t pass the output of initVM() to the parallelized function because this object is not pickable. So I can’t use the function AttachCurrentThread(). I am not very familiar with multi threading and multiprocessing so I don’t know if there is a way to get over this problem.
I also use openjdk 8.
I will consider switching to multithreading instead of multiprocessing or write my whole code in Java.
I’ve also been banging my head against multi CPU use over the last couple of days. What I’ve learned:
Multi Threading only benefits I/O bound tasks, like propagating many objects and writing the state to a file or database in each time step. Threads share the same memory space and you can “attach” the vm as mentioned above.
Multi Processing helps with cpu-bound tasks, but each process has an isolated memory space and requires that all data transferred in/out can be serialized (pickled). So you need to spool up a dedicated vm for each sub process.on init
Hi,ThomasJ!
Thanks for your post here,I’ve been stuck on this problem for a while. I just successfully debugged this method using multiprocessing. What’s the difference in implementation with mpi4py? Is there any difference in efficiency between the two methods?
mpi4py looks super interesting - thank you for sharing! From what I can tell it does not require separate java vm for each process which should be a lot more efficient.
@AstroL
I am far from being an expert on the topic, but from my little experience, here is the main difference I see :
multiprocessing is a python library that allowes you to divide a task on several processors and that provides some tools for communication between the processes.
MPI allowes you to do the same thing but it is not specific to python. When you run your python script with MPI, you must specify the number of processes you want to use (mpiexec -n 4 python myscript.py for instance). The script will be read by each process independently. mpi4py is a python library that allowes you to control MPI from your python script (communication) and to have a different behaviour depending on the process number (rank) using if rank == i: ... for instance.
@Sydra
I am glad to know that it is interesting to you!
From what I can tell it does not require separate java vm for each process which should be a lot more efficient.
Are you sure of that ? I think that you need to have separate java vm for each process. Can you provide more details on what makes you think that ?
The documentation for mpi4py is very good. You can check the tutorials that will help you understand how it works.
I’ve admittedly not have time to try this yet. For context, I’m using orekit as the engine of a space communications network simulation. As the number of spacecraft / locations goes up, it becomes computationally challenging.
Multi-threading didn’t help, and using multiprocess on an 8 core machine was actually slower than running single process due to the overhead of serializing the state data to pass it between processes.
I haven’t tried GPU yet (via CUDA) and MPI is still on my list too. I’ll report back when I had a chance to play with it …
Thanks for the relpy.
I’ve tested the multi process method in @Sydra’s post, it seems well for me.
I’ve also tested mpi4py, but something went wrong.Did you get this kind of problems?