Creating Normalisation Factors
==============================

Normalisation may be important when absolute values need to be found,
because in this case comparisons are done between two temperatures
or two other parameters, for the same sample, measured at two different times.

Otherwise if the intensity normalisation vary slowly,
and you search relative constants, it may not
be a problem when variation are small during the sweeping of a single spot.
(Tds2el fits separately the same hkl which has been swept two or more times during a scan).

When normalisation is important, we need a two columns file : first column
the name of the file; second column a number.
Rescale th is number not far from 1 so that you dont loose  the  idea
of the detector signal during the following analysis phases.

This file must be provided by you because metadata format, if any,
is different from setup to setup. Sometimes one uses a measurement,
sometimes the intensity of the beam, sometimes nothing at all.
When something is written each one uses a different name.

We are going to produce such a file based on a time
indication that we found in the files header.
If you dont have even that indication, than you better preserve
the file time-stamp when copying or tranfert data and use that.

We are going to use beam intensity of the Esrf machine, that
we can retrieve on rnice8 by using hdb_query ::

  /segfs/tango/release/java/scripts/hdb_query

and then select the proper informations

  .. image :: cassandra.png  

You select Accelerator->sr->c-ct->1->current, then you ask a table, choose a period start-end that covers your experiment, click on *perform Search*, you display the table tab and finally you save on a file, let\'s call it INTENSITY.dat. The produced files is like this ::
  
    # File generated from hdbviewer application
    #
    HDB Date	HDB Time	sr/d-ct/1/current (mA)	
    01/02/2017	05:47:36	191.95052130407686	
    01/02/2017	05:47:41	191.94287801288968	
    01/02/2017	05:47:46	191.93540346968973	
    01/02/2017	05:47:51	191.92684500481732	
    01/02/2017	05:47:56	191.9182165156779	
    01/02/2017	05:48:01	191.9105989850241
    01/02/2017	.......          ..............
    ...............................
    .................

    
Now we are going to produce the desired two columns file with
a python script ( should be easily adaptable to your case). We rescale by 170
because the experiment time lapse corresponded  more or less to this value ::

    import datetime
    import fabio
    import glob
    from scipy.interpolate import interp1d
    import time
    import string
    def get_timeintensity_interpolator(filename):
        times = []
        ints  = []    
        s=open(filename,"r").read()
        if len(s)<200:
            return string.atof(s)
        sl = string.split(s,"\n")
        for l in sl[3:]:
            if "Err" in l or "Advi" in l:
                continue
            l=string.split(l)
            if len(l)==0: break
            day,month, year = string.split(l[0],"/")
            hours,mins,secs = string.split(l[1],":")
            dt =  datetime.datetime(int(year), int(month), int(day), int(hours), int(mins), int(secs))
            s = time.mktime(dt.timetuple())
            # print s, float( l[2])
            times.append(s)
            ints.append(float( l[2] ))
        return interp1d(times, ints)

    interpolator = get_timeintensity_interpolator("INTENSITY.dat")
    output = open("normalisation.txt","w")
    fl = glob.glob("G19K45/set*cbf")
    for fn in fl:
        print fn
        im = fabio.open( fn,"r")
        tempo = im.getheader()["_array_data.header_contents"].split("\n")[1][2:]
        year, month,day,hours,mins,secs = tempo[:4], tempo[5:7], tempo[8:10],tempo[11:13], tempo[14:16], tempo[17:]
        dt =  time.mktime( datetime.datetime(int(year), int(month), int(day), int(hours), int(mins), int(float(secs))). timetuple()  )
        output.write("%s   %e\n"%(fn, interpolator(dt)/170.0) )


and you get the firl normalisation.txt ::

    G19K45/set1_0001p_02196.cbf   1.003309e+00
    G19K45/set1_0001p_03329.cbf   1.003275e+00
    G19K45/set1_0001p_00697.cbf   1.003347e+00
    G19K45/set1_0001p_03052.cbf   1.003279e+00
    G19K45/set1_0001p_01294.cbf   1.003335e+00
    G19K45/set1_0001p_02050.cbf   1.003315e+00
    G19K45/set1_0001p_00294.cbf   1.003358e+00
    G19K45/set1_0001p_00479.cbf   1.003355e+00
    G19K45/set1_0001p_00642.cbf   1.003351e+00
    G19K45/set1_0001p_01609.cbf   1.003325e+00
    G19K45/set1_0001p_01924.cbf   1.003320e+00
    G19K45/set1_0001p_01853.cbf   1.003320e+00
    G19K45/set1_0001p_02946.cbf   1.003283e+00
    G19K45/set1_0001p_02220.cbf   1.003309e+00
    ..........................................