I’m at IBM Research as the Community Lead for Open-Source Science (OSSci), a new initiative at NumFOCUS that aims to accelerate scientific research and discovery through better open source in science. OSSci was announced at the SciPy 2022 conference in July of last year, and our interest groups – focused on our initial topic areas chemistry, life sciences, and climate/sustainability – have been getting under way.

We are getting ready to launch our Reproducible Science IG. Maybe of interest to some of you here or people in your network.

Please check our Medium post and follow the link to the application form in case you’d like to get involved. Thanks!

A dice game

I believe this might be a good place to promote a project I am developing to assess to the quality of scientific predictions:

https://tedcrogers.wordpress.com

The problem of model lock-in
Model limitations, differences between specializations, and the problem of model lock-in.

With models we create a mathematical or logical representation of reality.

All models have a limitation

Example: I wanted to make a simulator for a small square on the sun. This to increase our understanding of the sun and potential problems.
I needed to model the flow of electrons, the flow of different ions, possible chemical reactions, possible nuclear reactions, possible electrical and magnetic influence, transfer and blocking of electrical and magnetic fields, induction currents, pressure and speed differences (like Stokes), gravity, ionizing radiation, heat transfers, etc.
And I wanted to use a particle system combined with a spacial system, to get the best simulation possible. So I could simulate a solar flare.

No this did not work. It is far too complex. Our simulations are not advanced enough. We are now just reaching the point that we can do water simulations with pretend-fire and pretend-foam. It may look good, but has not the accuracy to predict reality. The best sun simulators are using 2D and very simplified processes.

Differences between specializations

A major part of physics is modelling a small part of reality. and avoiding this complexity. They all use very simple models (mathematically) and extend from there. Like the Schrodinger formula in Quantum mechanics is based on a statistical formula to deal with infinite possibilities in a well defined environment. And that is why it is accurate in statistical predictions, but can not tell anything about a single prediction. Nor can it say much about how the electricity flows in a radio. Or how a ball bounces on a tennis court. Even though the graphs of the different processes can look very similar, like sinus waves.

And many different departments have specialized models to deal with certain problems. They have studied those problems thoroughly. But only within a certain environment, or certain context. And all of these have to be practical.
One math joke is that Engineers are using gravity= 10 m/s2 , PI=3, sin(x)=x. Because they would not need more accuracy. Not exactly true, but in practice our models are slightly off from reality anyway. Like: If you throw a ball in the air, you can not tell how high it goes, because your throw is never the same action. Nor are the weather conditions constant, etc.

False models

Any model, has practical limitations. But also have assumptions about the conditions and environment. And the most used models in a specialized field are extremely simplified. Not because they are correct, but because the are a lot easier to use.

But if you model too little, you also create false models. And this creates false ideas of reality. Or false ideas of a situation.

With the sun I noticed that many astronomers made assumptions about magnetism, that breaks with the electromagnetism that I know. They claimed that magnetic field lines were colliding with each other, producing bursts of energy. And in our earth's physical reality this is impossible. We have radios, electric-motors and all kinds of electromagnetic machinery. We have magnetic fields colliding all the time and they just add together with no implications at all. Magnetic field lines are also abstract representations of a continuous field, and have no physical meaning.
So something is wrong here: these astronomers were using an oversimplified model, and extended it far beyond its limitations. And added field-lines as a physical concept. It is already a complex model (Magnetohydrodynamics/HMD), and can be used in certain limited context as its inventor Alfven described. But the simplifications also gives some weird outcomes, and their predictions for solar flares are 1 million times off. Most astronomers know that something is wrong and MHD is often called "magic".
Still this is the dominant theory in mainstream solar physics.

Model Lock-in

But this problem is in all fields of science. Each specialization has its own home-grown models. Models that work well in the situations that they test for. Or when the test fails, they keep them, because they preferred those models the most.

And in every specialization of science I noticed that they have locked-in their preferred models. Psychology is full with such locked-in models. Some social studies have presented imaginary models even as facts. While advancement of science has always been the change of models and their related theories.

With physics we can see in the laboratory that certain models go wrong. And that is why we have advanced so much technologically. Yet even with physics the preferred models are often mixed with additional correction-models, instead of replacing one of the locked-in models. Even if they could exist side-by-side.
But with psychology there is almost no way to verify a certain model, and the outcome can also be influenced. The same is in big medicine, where the models and outcomes of tests are influenced by the need for profits.

What happens if a test fails?

And different specializations of science have different techniques to keep their preferred models. (And I apply them to my solar flare model)

1. Misuse of Authority (You are not an astronomer).
2. Misuse of overly-critical peer-review (Your criticism will not be published).
3. Claim it is coincidence or a fail of the test (Next time will show the prediction is correct).
4. Use personal attacks on the testers or scientists (You are a flat-earther).
5. Claim it is fake. (We see no problems)
6. The situation is special. (The sun is a special place)
7. You understand it wrong, you are too stupid. (The sun can only be understood by very very smart people)
8. Cancel culture (We ban you from this subreddit - really happened)
9. Trust us "(Astronomers will soon understand more)*
10. It can not be wrong. We always used it.
11. There is no other possibility.

What I wanted was a normal open discussion with clear data and clear science. Not all these logical fallacies. Whatever model is correct, the system is clearly broken.

How can we solve this

Any model that does not match with the tests is basically broken.

What are the limitations of the model? Where are the boundaries? What precision can be expected? Do we anywhere correct the data towards the model?

Is there a huge difference between the expectations and the reality in the field?

We need backtracking by observing unbiased reality. Especially when we have better observations or better data. Does the model still make sense? Can we correct the model? Can we drop the old/worse data? Is a different model better? And most importantly: can we accept criticism?

What if all the preferred models are completely wrong? Or what if we simply do not know?

From my understanding (as I have not read many of them), most thinkers tend to critique technology, data, and analytics from the idea of if this is good for society's soul or psyche or how it is transforming society.

Instead, I am after if the stated goals of these three concepts are actually useful. For example, I hope to go into marketing, and it is dominated by analytics. I am curious if analytics itself is even useful for the various goals it sets out to accomplish.

Another example, I recall skimming a text or article that crafted science without mathematics. In a way, it challenged the usefulness of mathematics (please do not take this basic analysis as serious, it's only a cursory thought stemming from something I skimmed years ago).

Any resources (texts, articles, videos, or even in-depth comments) are welcomed, and I appreciate your time!