Jscatter’s documentation

The aim of Jscatter is treatment of experimental data and models:

Jscatter Logo

  • Reading and analyzing experimental data with associated attributes as temperature, wavevector, comment, ….

  • Multidimensional fitting taking attributes into account.

  • Providing useful models for neutron and X-ray scattering form factors, structure factors and dynamic models (quasi elastic neutron scattering) and other topics.

  • Simplified plotting with paper ready quality.

  • Easy model building for non programmers.

  • Python scripts/Jupyter Notebooks to document data evaluation and modelling.

Binder citation install license pyversion Read the Docs Beginners Guide

Main concept

Exemplary functions:

Physical equations and useful formulas as quadrature of vector functions

  • scatteringLengthDensityCalc() -> Electron density, coh and inc neutron scattering length, mass.

  • smear() -> Smearing enabling simultaneous fits of differently smeared SANS/SAXS data.

  • desmear() -> Desmearing according to the Lake algorithm for the above.

See formel and smallanglescattering (sas)

Particle Formfactors

See formfactor (ff)

Particle Structurefactors

  • RMSA() -> Rescaled MSA structure factor for dilute charged colloidal dispersions.

  • twoYukawa() -> Structure factor for a two Yukawa potential in mean spherical approximation.

  • hydrodynamicFunct() -> Hydrodynamic function from hydrodynamic pair interaction.

See structurefactor (sf)

Dynamics

See dynamic

biomacromoleules

See Biomacromolecules (bio)

How to use Jscatter

see Examples and Beginners Guide / Help or try Jscatter live at Binder .

# import jscatter and numpy
import numpy as np
import jscatter as js

# read the data (16 dataArrays) with attributes as q, Dtrans .... into dataList
i5 = js.dL(js.examples.datapath + '/iqt_1hho.dat')

# define a model for the fit
diffusion = lambda A, D, t, wavevector, e=0: A*np.exp(-wavevector**2*D*t) + e

# do the fit
# single valued start parameters are the same for all 16 dataArrays
# list start parameters [...] indicate independent fitting for dataArrays
# the command line shows progress and the final result, which is found in i5.lastfit
i5.fit(model=diffusion,                         # the fit function
       freepar={'D': [0.08], 'A': 0.98},        # free start parameters
       fixpar={'e': 0.0},                       # fixed parameters
       mapNames={'t': 'X', 'wavevector': 'q'})  # map names from model to data

# open plot with results and residuals
i5.showlastErrPlot(yscale='l')

# open a plot with fixed size and plot data and fit result
p = js.grace(1.2, 0.8)
# plot the data with Q values in legend as symbols
p.plot(i5, symbol=[-1, 0.4, -1], legend='Q=$q')
# plot fit results in lastfit as lines without symbol or legend
p.plot(i5.lastfit, symbol=0, line=[1, 1, -1])

# pretty up if needed
p.yaxis(min=0.02, max=1.1, scale='log', charsize=1.5, label='I(Q,t)/I(Q,0)')
p.xaxis(min=0, max=130, charsize=1.5, label='t / ns')
p.legend(x=110, y=0.9, charsize=1)
p.title('I(Q,t) as measured by Neutron Spinecho Spectroscopy', size=1.3)
p.text('for diffusion a single exp. decay', x=60, y=0.35, rot=360 - 20, color=4)
p.text(r'f(t)=A*e\S-Q\S2\N\SDt', x=100, y=0.025, rot=0, charsize=1.5)
Picture about diffusion fit

Shortcuts:

import jscatter as js
js.showDoc()                  # Show html documentation in browser
exampledA=js.dA('test.dat')   # shortcut to create dataArray from file
exampledL=js.dL('test.dat')   # shortcut to create dataList from file
p=js.grace()                  # create plot in XmGrace
p=js.mplot()                  # create plot in matplotlib
p.plot(exampledL)             # plot the read dataList

jscatter.usempl(*args)

Switch between using mpl and grace in dataArray/dataList fitting.

jscatter.headless(*args)

Switch to headless mode (NO-GUI) to work on a cluster.

jscatter.version

Jscatter version is changed in pyproject.toml
If not otherwise stated in the files:

written by Ralf Biehl at the Forschungszentrum Jülich ,
Jülich Center for Neutron Science 1 and Institute of Complex Systems 1
Jscatter is a program to read, analyse and plot data
Copyright (C) 2015-2021 Ralf Biehl

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Jscatter package contents

Installation

Dependencies

  • numpy, scipy -> Mandatory, automatically installed by pip

  • Cython -> Mandatory, automatically installed by pip

  • Pillow -> Mandatory, automatic install by pip, for reading of SAXS images

  • matplotlib -> Mandatory, automatic install by pip, for 3D plots and on Windows

  • Pillow, -> Mandatory, automatic install by pip, image reading

  • fabio, -> Mandatory, automatic install by pip, reading two-dimensional X-ray detectors

  • MDAnalysis -> Mandatory, automatic install by pip, analyze trajectories from MD simulations

  • pdb2pqr -> Mandatory, automatic install by pip

  • emcee -> Mandatory, automatic install by pip, Baysian analysis

  • nglview -> Optional, automatic install by pip, interactively view molecular structures

  • Ipython -> Optional, for convenience as a powerful python shell

  • gfortran -> Optional, without some functions dont work or use a slower python version

  • xmgrace -> Optional, preferred plotting on Unix like (use matplotlib on Windows)

Installation of gfortran/xmgrace may need root privileges. Use “sudo” on Linux and MacOS if needed.

Pip installation/upgrade

As user (without root access) this will install to your home directory (by default to path ~/.local/). Prepend sudo if it should install to the system python (optional).

python -m pip install jscatter

It is assumed that python3 is your default Python environment and dependencies are installed by a system administrator. You can check python version typing python in the terminal and look at the first lines with version information.

To install dependencies see below.

Check Troubleshooting and tips for help if this is not the case or jscatter is not found after installation.

To upgrade to latest version

python-m pip install jscatter --upgrade

Install from a local repository (development versions):

python -m pip install jscatter --upgrade --pre --find-links /where/the/file/is/saved

options
--user       : Install in user directory (folder defined by PYTHONUSERBASE or the default ~/.local)
--find-links : look in the given path for package links e.g development releases
--upgrade    : to install upgrades also for dependencies
--pre        : to install also development versions

Installation directly from from git repository branch. For different branch use branch name. This is only for developpers. dev versions are under development! (You may need a user account)

python -m pip install git+https://gitlab.com/biehl/jscatter.git@dev

Linux

  • Ubuntu, all Debian related

    sudo apt install build-essential  # install gcc and more
sudo apt install cmake gfortran grace  # or python3-matplotlib e.g. Debian,Fedora, RedHat
python -m pip install ipython
python -m pip install jscatter

  • RockyLinux, Suse, Fedora … do same as above but with yum/zypper…

  • Manjaro Linux (yaourt asks for permission as root or prepend sudo as above)

    # install these for fortran
pamac install gcc-fortran cmake
# install xmgrace (only found in AUR), fonts are needed for the interface, fonts are loaded after restart
pamac install xorg-fonts-75dpi xorg-fonts-100dpi pdflib-lite
pamac install grace
# tk might be missing for tkinter as matplotlib backend
pamac install tk
python -m pip install ipython cython
python -m pip install jscatter

  • CONTIN in DLS module (Only if needed).

    See DLS module documentation for details how to get and compile the original fortran code.

MacOs (Intel + Arm)

If you have a preinstalled gcc compiler you need the Xcode Command Line Tools. A command like gcc may trigger the installation. Manually do

xcode-select --install

All the following can be done without brew installing in a different way. But ist simple if you are your own admin.

Homebrew is a MacOs package manager. The following should work on Catalina and BigSur and Sonoma. Tested on Intel an Arm (M3)

Install Homebrew first as given on their web page (see Homebrew). Homebrew installs things to [Arm]/opt/homebrew, [Intel]/usr/local so you don’t need sudo permissions.

[Intel] Append the following line to your .zshrc file and restart the terminal:

# [Intel] set Path to point to homebrew
export PATH=/usr/local/bin:/usr/local/share/python:$PATH
# [Intel] XQuartz needs this to use flatNamespace correctly with grace (error was something with Vendorshell...)
export DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace

You need Python and more. Here I install from Homebrew which is easier. Other ways might work too. Install XQuartz, gcc and xmgrace from Homebrew:

# install XQuartz from homebrew cask (NOT from the AppStore, because of above setting)
brew cask install xquartz
# install xmgrace and gfortran (included in gcc)
brew install gcc grace cmake
# maybe its necessary to relink if old links from previous install are there
brew link --overwrite gcc
brew link --overwrite grace

For newer MacOs Python is not installed:

# install python from homebrew (by default python3 and be called by `python3`)
brew install python

Install needed things via pip.

# then use pip
python3 -m pip install ipython
# dependencies are installed automatically
python3 -m pip install jscatter
# check which python is used
which python3

Troubles - for xquartz maybe a restart is needed to set all things right. - If the compiler is not found set it explicitly. See Troubleshooting below.

  • [Intel] If you have trouble like missing xcrun .. you might have to re-register it or update it to the latest version. Try xcode-select –install.

  • In MacOs [Intel] (e.g. Catalina) you may get an error “stdio.h not found” which is related to the missing /usr/local/include folder as decided by Apple. Add to .zshrc or corresponding to include this path in CPATH:

    export CPATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/

Windows

Windows Subsystem for Linux (WSL) with Ubuntu way

A new way is based on Windows Subsystem for Linux (WSL), which “lets developers run GNU/Linux environment – including most command-line tools, utilities, and applications – directly on Windows, unmodified, without the overhead of a virtual machine.” (See WSL , use WSL2)

This allows to use any Linux application including XmGrace or Linux editors. You work on the same filesystem as your Windows user account without the need of syncing folders or using a shared folder from a virtual machine. Also gfortran compiled code is working.

The basic procedure is to activate WSL, install Linux and Xserver, add your needed Linux software in the usual way.

The following is for Windows 10. For Windows 11 its simpler (Ubuntu is installed by default and the Xserver is working already)

  • Install WSL (Please check for the update to WSL2 on following site)

    See https://learn.microsoft.com/de-de/windows/wsl/install On Windows11 Ubuntu is installed and it asks already for username and password.

  • Windows10: install a linux distribution of choice from Windows store (tested with Ubuntu 20.04). (The install is per user, so do it under your user account, NOT as administrator)

    Open a terminal, start bash and set user and password for Linux. (See https://docs.microsoft.com/en-us/windows/wsl/initialize-distro )

  • Windows10: For graphical applications: Install Xserver VcXsrv (or an other Xserver).

    See e.g. https://aalonso.dev/blog/how-to-use-gui-apps-in-wsl2-forwarding-x-server-cdj or https://techcommunity.microsoft.com/t5/windows-dev-appconsult/running-wsl-gui-apps-on-windows-10/ba-p/1493242. Download from https://sourceforge.net/projects/vcxsrv/ . Install (start always manually or add to Windows autostart).

    For WSL2 :

    • If you start vcxsrv on Windows check in the last option window that “Disable access control” is checked to allow connections from WSL. If you save the configuration it might remind the settings.

    • Open the Windows Firewall for the Xserver: When starting the Xserver the first time allow public and private Networks. If you missed this, open Windows Security -> Firewall & network protection -> Allow an app through firewall -> make sure VcXsrv has both public and private allowed (checked).

      If you dont find VcXsrv you need to add the executable with path like C:\programm files\vcxsrv\vcxsrv.exe and allow public and private access.

  • Windows10: Configure bash to use the local X server

    Add this to .bashrc at the end dependent on using WSL1 or WSL2:

    export DISPLAY=localhost:0.0 # WSL1
export DISPLAY=$(grep -m 1 nameserver /etc/resolv.conf | awk '{print $2}'):0.0 # WSL2

    or run this inside bash to append it to .bashrc:

    # for WSL1
echo "export DISPLAY=localhost:0.0" >> ~/.bashrc
# for WSL2
echo "export DISPLAY=\$(grep -m 1 nameserver /etc/resolv.conf | awk '{print $2}'):0.0" >> ~/.bashrc

    Restart bash or load again .bashrc by source ~/.bashrc

  • Windows10/11 Install xmgrace, python3 and Jscatter in Linux subsystem (on newer systems some is already installed)

    open terminal, start bash

    sudo apt update && upgrade                                   # upgrade linux
sudo apt install python3 python3-pip ipython3 gfortran gcc   # python, ipython and pip3
sudo apt-get install cmake numpy xmgrace python-tk           #
python -m pip install cython
python -m pip install jscatter                               # jscatter, scipy, Pillow...

Run test and example.

  • Advice to make the shell more comportable

    Add these lines to .bashrc file (open a wsl terminal and nano ~/.bashrc)

    It defines a shortcut for the Windows User directory and goes there. (replace username withyours)

    # my adons
alias cdhome="cd /mnt/c/Users/username"
cdhome

Anaconda way

Anaconda is a python distribution as alternative with numpy, scipy, matplotlib, Ipython preinstalled. Need of sudo depends on how Anaconda was installed (root or user). Maybe the matplotlib backend needs to be configured on Windows to work properly. XmGrace is not working on native Windows.

And there was more to adjust, when i stopped waisting my time. In my testcase it was a pain, but test and examples work (no Xmgrace, no fortran).

I strongly advise to use the WSL way from above, as it is easier to install, update and use. And even gfortran is working in WSL!!

# install jscatter on working anaconda environment
python -m pip install jscatter

Jupyter Notebook

Jscatter works on Jupyter Notebooks respectively Jupyter-lab. You may try Jscatter live at Binder.

The example notebooks are included in the example subfolder of your Jscatter installation or can be downloaded from Gitlab/Jscatter .

To install jscatter on a server Jupyter installation (Jscatter not preinstalled) prepend this on your script

import sys
# install jscatter as user in the current Jupyter kernel
!{sys.executable} -m pip install jscatter

There is some trouble with inline plots (which are not interactive and cannot be updated) dependent on the installed backend. This is related to the used matplotlib backend not to Jscatter.

Use this (before importing anything else) to get interactive plots inline.

%matplotlib widget

Then import Jscatter.

import jscatter as js

js.usempl(True)  # use matplotlib
js.usempl(False) # use grace on your computer with xmgrace in external window.

If you work over http on a server (usual no display of Xwindows applications) use the same to switch to matplotlib.

Using a cluster/headless mode

For long running scripts or fitting one might want to use a cluster.

Cluster allow often no graphical output (as they are used e.g. trough ssh running a script.). To run your script as usual (after testing on your PC) Jscatter implements a headless mode which can be activated using:

import jscatter as js
js.headless(True)

In headless mode all errPlots are saved automatically to lastErrPlot.agr and open matplotlib plots are saved as lastopenedplots_i.png if js.mpl.show() is used. Both allow to retrieve an updated graphical output as stored image instead of updated windows.

Please save important plots in the usual way to store them with a useful name. Additional any data output should be saved in the usual way. Xmgrace is advantageous as .agr format allows later change of the plots including rescaling.

Examples are also adapted and saved to jscatterExamples/example_name/ under the current working directory.

For experts:

Xmgrace uses gracebat and matplotlib uses the non-interactive backend “Agg”. gracebat needs to be installed.

For matplotlib create figures using fig=js.mpl.mplot or fig=matplotlib.figure.Figure and later save by fig.savefig. Dont use pyplot as it generates errors in headless mode without interactive loop.

To switch back it is best to restart ipython (at least for matplotlib).

Testing

You can test basic functionality of jscatter after installation:

import jscatter as js
js.test.doTest()
#basic graphics and fitting
js.examples.runExample('example_SinusoidalFitting.py')
The Example shows :

  • 3 sine fit plots with one sine

  • a fit plot with 5 sine curves fitted simultaneous

  • a simple plot with 5 points ( phase against Amplitude of the 5 sines)

Troubleshooting and tips

  • The jscatter version can be checked by js.version .

  • For testing and playing around without disturbing your productive environment you migth use https://virtualenv.pypa.io .

    virtualenv is a tool to create isolated Python environments.

  • In case of problems use -v switch to get more output during install.

    python -m pip install jscatter -v

  • If the gcc or gfortran compiler is not found (maybe on a Mac with homebrew) or a different compiler is tried set the compiler explicitly before installing jscatter. Default is gcc/gfortran, below on a i86 MAC installed from homebrew.

    # to use installed gcc compiler version 12
export CC=/usr/local/bin/gcc-12
export FC=/usr/local/bin/gfortran-12

  • If jscatter is not found (or an old version) after installation/upgrade then it might be a problem of having multiple versions of Python on your system. Starting pip a different version than starting python is the reason.

    On older systems, using pip -> pip3 uses the python3 version of pip if the old python2 is your default .

    Using instead of pip -> python -m pip in the all above commands calls the correct pip version in any case (the Python version started with python). This is for example needed if you have a system python which should not be touched and a second python for users.

    E.g. like this to install also ipython in the same python environment

    python -m pip install ipython
python -m pip install jscatter

  • If xmgrace is not found by jscatter (`module 'jscatter' has no attribute 'grace'`) the path to the executable may be not on your PATH variable.

    Check this by calling xmgrace in a shell, if it is started it should work.

    If xmgrace is not starting, add accordingly the installation path to your PATH variable. For e.g. bash add in your .bashrc (for zsh on MacOs .zshrc) on my CentOS

    export PATH=/usr/local/bin:$(PATH)

    Restart the shell after changing.

  • To open .agr files by klicking add a new file association e.g. to KDE.

    • In SystemSettings/FileAssociations add a new type xmgrace.

    • Inside this add FilenamePatterns ‘*.agr’ and similar.

    • In ‘ApplicationPreferenceOrder’ add xmgrace and edit this new application :

      In General: edit name to “xmgrace” (keep it). In Application: edit name to “xmgracefree” and command to “xmgrace -free”. This will open files in free floating size format.

      In ApplicationPreferenceOrder add again application xmgrace (no changes) The second opens files in fixed size format (no ‘-free’).

Indices and tables