README.md 11.7 KB
Newer Older
1
# Metadata Manager and Experiment
2

3 4 5 6 7 8
- [Metadata Manager and Experiment](#Metadata-Manager-and-Experiment)
  - [Dataset Location](#Dataset-Location)
    - [DataFolderPattern](#DataFolderPattern)
    - [Specific dataset Location](#Specific-dataset-Location)
  - [Requirements](#Requirements)
    - [Debian 6](#Debian-6)
9 10
  - [Configuration](#configuration)
    - [Tango](#tango)
11 12 13 14
      - [Using automatic tool to generate tango properties](#Using-automatic-tool-to-generate-tango-properties)
    - [ICAT Reader](#ICAT-Reader)
    - [ICAT+](#ICAT)
    - [Logging with Graylog](#Logging-with-Graylog)
15
    - [Sum Up](#sum-up)
16 17 18 19 20 21 22 23
  - [Build](#Build)
    - [Packaging](#Packaging)
    - [Deploy](#Deploy)
  - [Elogbook](#Elogbook)
    - [Requirements](#Requirements-1)
    - [Usage](#Usage)
      - [Python](#Python)
  - [Simulation](#Simulation)
24

25
7. [Simulation](#simulation)
26

27
## Dataset Location
28

29
First Metadata Manager needs to be configured with some parameters. Next parameters are mandatory: 
30
- **dataFolderPattern**: pattern that will be replaced by the values from parameters to determine the dataset location. For instance: {dataRoot}/{proposal}/{beamlineID}/{sampleName}/{scanName}
31
- **globalHDFfiles**: patterns that will be replaced by the values from parameters to determine the location where global HDF5 file will be stored. For instance: {dataRoot}/{proposal}/{beamlineID}/{proposal}-{beamlineID}.h5
Staffan Ohlsson's avatar
Staffan Ohlsson committed
32
- **beamlineId**: name of the beamline. For instance: id30a3, bm31
33

Staffan Ohlsson's avatar
Staffan Ohlsson committed
34
To be Changed!!
35
![jive](documentation/jive-screenshot-manager-properties.png)
36

37 38 39
If a datasetParentFolder is specified will be used to determine the final dataset location otherwise the dataFolderPattern will be used:


40 41 42
![datasetfolder.png](documentation/flowchart/datasetfolder.png)


43
### DataFolderPattern
44

45
This flow chart explains all steps needed to store a dataset.
46

47 48
![flowchart](documentation/flowchart/MetadataExperimentandMetadataManagerFlowchart.png)

49 50 51 52 53 54 55 56 57 58 59 60 61 62
In the previous example the final dataset folder will be determined by the dataFolderPattern.

### Specific dataset Location

It is possible to force the dataset location by using the method setDatasetParentLocation. This method receives a String as parameter with the folder of the dataset's parent.

Example:
```
metadataManager.setDatasetParentLocation("/data/visitor/MA1234/id01/12042018")
```
In this example the final dataset's folder will be /data/visitor/MA1234/id01/12042018/{scanName}

![setfolder](documentation/flowchart/setFolder.png)

63
In order to implement such functionality 3 new methods have been implemented:
64 65

1. **setDatasetParentLocation** is optional and can be composed by:
66
A pattern which keys will be replaced. For instance: {dataRoot}/{proposal}/Jonh/{sampleName}/20180418
Staffan Ohlsson's avatar
Staffan Ohlsson committed
67
An absolute path: /data/visitor/MA1234/id01/Johh/12042018
68

69 70 71
2. **getDatasetParentLocation** returns the value of datasetParentLocation
3. **clearDatasetParentLocation** sets the value of datasetParentLocation to null then dataFolderPatter will be used instead.

72
In both cases these folder locations point to the **parent** folder of the dataset
73

74 75
## Requirements

76
MetadataManager has got as **optional** dependency graypy (https://pypi.python.org/pypi/graypy) and requests
77

78 79 80
Requests is a requirement for sending notifications to the **elogbook** and graypy is required to use **Graylog**

They can be installed:
81 82
```
easy_install graypy
83
easy_install requests
84 85 86 87
```
or
```
pip install -U graypy
88
pip install requests
89 90
```

91
### Debian 6
92

93 94 95 96 97 98 99 100
Recommended solution is:

First try to use requests as supplied by debian. If you run into issues,
then try to install the packages downloaded  (they install a much newer
version).

The module graypy can be easily installed in any case by downloading the
source package (pip install or python setup.py whatever works).
101

102 103


104 105
## Configuration

106 107
### Tango

Staffan Ohlsson's avatar
Staffan Ohlsson committed
108
#### Using automatic tool to generate tango properties
109

110 111
There is a script that can be found in /scripts/GenerateMetaDataResource folder that generates the right starting configuration for a beamline. 
Usage:
112
```
113
Usage: GenerateMetaDataResources.py -b BL_NAME [options]
114

115
MetaData TANGO DEVICE SERVER RESOURCES GENERATOR
116

117 118 119 120 121 122 123 124 125
Options:
  -h, --help            show this help message and exit
  -b BL_NAME, --beamline=BL_NAME
                        the beam line name
  -m MEMBER_NAME, --member_name=MEMBER_NAME
                        the member part of the three part device
                        (domain/family/member)
  -p PERSONAL_NAME, --personal_name=PERSONAL_NAME
                        the personal name of the Device Server
Staffan Ohlsson's avatar
Staffan Ohlsson committed
126

127 128
```

129
Example:
Staffan Ohlsson's avatar
Staffan Ohlsson committed
130
```
131
GenerateMetaDataResources.py -b id99 
Staffan Ohlsson's avatar
Staffan Ohlsson committed
132
```
133

134
The use in Jive -> File/Load property file to load the generated file.
135

136
Example of generated files:
137
```
138 139
MetadataManager_ID99.tango
MetaExperiment_ID99.tango
140 141
```

142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160
### ICAT Reader

From version > release-v1.63 MetadataManager has got a new function called CheckParameters. This function compares the list of parameters with the ICAT database and will display a message if there are parameters not support on the currect ICAT configuration.

For instance:

```
----------------------------------------------------
Command: id00/metadata/mgr/CheckParameters
Duration: 143 msec
Output argument(s) :
----------------------------------------------------


[ERROR] 2 parameters are unknown by ICAT. Ingestion will fail.
[Parameter] myParameters
[Parameter] myLabels
```

161 162 163 164 165 166 167 168 169 170
In order to configure the access from the MetadataManager to ICAT is necessary to set up some parameters in the class properties:
- AuthenticationPlugin: db
- username: reader
- password:  reader
- port: 443
- server: icat.esrf.fr


![icat-properties.png](documentation/icat-properties.png)

171 172 173 174 175 176 177
### ICAT+

MetadataManager can send notifications to ICAT+. For doing so, some parameters need to be added into the class properties:
```
OBJ_PROPERTY:API_KEY: elogbook-be70ac55-fd08-4840-9b29-b73262958ca8
OBJ_PROPERTY:icatplus_server: "http://lindemaria:8000"
```
178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211
### Sum Up

You need to define as class parameters:

Non-Debian6
```
API_KEY: elogbook-be70ac55-fd08-4840-9b29-b73262958ca8
authenticationPlugin: "db"
icatplus_server: "https://icatplus.esrf.fr"
password: "reader"
port: "443"
queueName: "/queue/icatIngest"
queueURLs: bcu-mq-01.esrf.fr:61613,\ 
           bcu-mq-02.esrf.fr:61613
server: "icat.esrf.fr"
username: "reader"

```

For Debian6
```
API_KEY: elogbook-be70ac55-fd08-4840-9b29-b73262958ca8
authenticationPlugin: "db"
icatplus_server: "https://icatplus.esrf.fr:8443"
password: "reader"
port: "443"
queueName: "/queue/icatIngest"
queueURLs: bcu-mq-01.esrf.fr:61613,\ 
           bcu-mq-02.esrf.fr:61613
server: "icat.esrf.fr"
username: "reader"

```

212

213
### Logging with Graylog
214 215 216

Graylog configuration is hardcoded in MetadataManager class. To be moved to a configuration file.

217
## Build
218

219
### Packaging
220 221

Build is done by using the next command that will also package the component in a .zip file.
222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237
```
mvn package
```

This will create a distrib.zip with the distribution:
```
lindemaria:workspaces/icat/tango-metadata % mvn package
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Tango MetadataManager 1.63
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:regex-property (pythonVersion) @ MetadataManager ---
[INFO] No match to regex '-SNAPSHOT' found in '1.63'. The initial value '1.63' is left as-is...

238 239 240
[..................]


241 242 243 244 245 246 247 248 249 250 251 252 253
[INFO] 
[INFO] --- maven-assembly-plugin:2.4:single (make-python-distrib) @ MetadataManager ---
[INFO] Reading assembly descriptor: /users/demariaa/Software/workspaces/icat/tango-metadata/src/assembly/distrib.xml
[INFO] Building zip: /users/demariaa/Software/workspaces/icat/tango-metadata/target/MetadataManager-1.63-distrib.zip
[INFO] Building tar: /users/demariaa/Software/workspaces/icat/tango-metadata/target/MetadataManager-1.63-distrib.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.222s
[INFO] Finished at: Thu May 18 13:50:06 CEST 2017
[INFO] Final Memory: 13M/303M

```
254 255


256
### Deploy
257

Staffan Ohlsson's avatar
Staffan Ohlsson committed
258
1.- ssh -X blissdb8 as yourself
259 260

```
Staffan Ohlsson's avatar
Staffan Ohlsson committed
261
lindemaria:~ % ssh -X demariaa@blissdb8
262 263 264 265 266 267 268
```

2.- Unzip MetadataManager-1.63-distrib.zip into /segfs/bliss/source/tango/server/metadata
```
esrflinux2-1a:server/metadata/MetadataManager-1.63 % ls -ltra
total 28
drwxr-xr-x  2 demariaa soft 4096 Mar  3  2016 data
Staffan Ohlsson's avatar
Staffan Ohlsson committed
269
drwxr-xr-x  2 demariaa soft 4096 May 18 13:47 demo
270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285
drwxr-xr-x  4 demariaa soft 4096 May 18 13:47 MetadataManager
-rw-r--r--  1 demariaa soft  194 May 18 13:50 setup.py
drwxr-xr-x  2 demariaa soft 4096 May 18 13:50 bin
drwxr-xr-x  6 demariaa soft 4096 May 18 13:50 .
drwxrwxr-x 15    80138 soft 4096 May 18 14:07 ..

```
3.- Make an installer with blissbuilder

Open blissbuilder:
```
/segfs/bliss/bin/blissbuilder
```

Browse into the project list Control -> Tango -> Server -> Metadata and click next

Alejandro De Maria Antolinos's avatar
Alejandro De Maria Antolinos committed
286

287 288
Select the files and follow the instruccions

Alejandro De Maria Antolinos's avatar
Alejandro De Maria Antolinos committed
289
## Elogbook
290

Alejandro De Maria Antolinos's avatar
Alejandro De Maria Antolinos committed
291 292 293 294
### Requirements

Go the the requirements section where dependencies are explained: [Requirements](#requirements)

295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376
### Usage

#### Python

Example:
```
#!/usr/bin/env python

"""A simple client for MetadataManager and MetaExperiment
"""
import time
import os
import sys
import logging
import PyTango.client
from time import gmtime, strftime

class MetadataManagerClient(object):

    metadataManager = None
    metaExperiment = None

    """
    A client for the MetadataManager and MetaExperiment tango Devices

    Attributes:
        name: name of the tango device. Example: 'id21/metadata/ingest'
    """
    def __init__(self, metadataManagerName, metaExperimentName):
        """
    	Return a MetadataManagerClient object whose metadataManagerName is *metadataManagerName*
    	and metaExperimentName is *metaExperimentName*
    	"""
        
        self.proposal = None                
    	if metadataManagerName:
            self.metadataManagerName = metadataManagerName
        if metaExperimentName:
    		self.metaExperimentName = metaExperimentName

        print('MetadataManager: %s' % metadataManagerName)
        print('MetaExperiment: %s' % metaExperimentName)

       	""" Tango Devices instances """	 
        try:
            MetadataManagerClient.metadataManager = PyTango.client.Device(self.metadataManagerName)
            MetadataManagerClient.metaExperiment = PyTango.client.Device(self.metaExperimentName)
        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise


    ''' Set proposal should be done before stting the data root '''
    def setProposal(self, proposal):
        try:
            MetadataManagerClient.metaExperiment.proposal = proposal
            self.proposal = proposal
        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
  
    def notifyInfo(self, message):
        MetadataManagerClient.metadataManager.notifyInfo(message)

    def notifyError(self, message):
        MetadataManagerClient.metadataManager.notifyError(message)

    def notifyDebug(self, message):
        MetadataManagerClient.metadataManager.notifyDebug(message)

if __name__ == '__main__':
    metadataManagerName = 'id00/metadata/mgr'
    metaExperimentName  = 'id00/metadata/exp'
    client = MetadataManagerClient(metadataManagerName, metaExperimentName)
                   
    client.setProposal('ID000000') 
    client.notifyInfo("This is a info") 
    client.notifyError("This is a error")
    client.notifyDebug("This is debug")


```
377 378


379
## Simulation
380

381
MetaExperiment and MetadataManager can be run in Simulation mode. It means that it will not try to access to the ActiveMQ server.
382

383 384
In order to activate the simulation mode someone just need to empty the queueURLs parameter
![icat-properties.png](documentation/classStatic.png)
385