user.rst 29 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714
  1. TaskManager User Guide
  2. ======================
  3. .. contents:: Table of Contents
  4. :depth: 2
  5. Installation
  6. ------------
  7. To install the GeoServer Task Manager extension:
  8. 1. Download the extension from the :website:`GeoServer Download
  9. Page <download>` release page: :download_community:`taskmanager-core`. For `S3 support <#s3-file-service>`__, also
  10. install the plugin :download_community:`taskmanager-s3`
  11. 2. Extract this file and place the JARs in :file:`WEB-INF/lib`.
  12. 3. Perform any configuration required by your servlet container, and
  13. then restart. On startup, Task Manager will create a configuration
  14. directory :file:`taskmanager` in the GeoServer Data Directory. You will
  15. be able to see the Task Manager configuration pages from the
  16. GeoServer WebGUI menu.
  17. Server Configuration
  18. --------------------
  19. Configuration Database & Clustering
  20. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  21. By default, Task Manager will create a H2 database in its configuration
  22. directory. This can be easily changed to any JDBC resource via the
  23. ``taskmanager.properties`` file.
  24. The configuration directory also contains a Spring configuration file
  25. called ``taskManager-applicationContext.xml`` which allows more advanced
  26. configuration.
  27. TaskManager uses `Quartz Scheduler <http://www.quartz-scheduler.org>`__.
  28. If you are running Task Manager in a clustered environment, you must
  29. configure Quartz to use a database as well as Task Manager. See the
  30. commented block in the Spring configuration and the `Quartz documentation <http://www.quartz-scheduler.org/documentation/quartz-2.3.0/configuration/ConfigJDBCJobStoreClustering.html>`__
  31. for further instructions. SQL Scripts to create the required database structures for Quartz can be found `here <https://github.com/quartz-scheduler/quartz/tree/quartz-2.3.x/quartz-core/src/main/resources/org/quartz/impl/jdbcjobstore>`__. Task Manager can create its database automatically, or alternatively :download:`this script <taskmanager.sql>` can be used (note: the script was made for postgresql. For any other DBMS, the script might need to be modified, or alternatively, the database could be automatically created in a development environment and then copied for a production environment). It is fine to use a single database both for Quartz and Task Manager.
  32. Furthermore, a property should be added to the
  33. ``taskmanager.properties`` file each of the nodes except for one:
  34. ``batchJobService.init=false``. This is necessary because otherwise all
  35. of the nodes will attempt to load all of the same batches in to the
  36. clustered quartz database at the same time at start-up, which is likely
  37. to cause issues. This initialisation needs to happen only once for the
  38. entire cluster.
  39. .. _taskmanager_user_databases:
  40. Databases
  41. ~~~~~~~~~
  42. Task Manager allows any number of databases to be used both as sources
  43. and targets for data transfer operations. These are configured via the
  44. Spring configuration file. Currently only PostGIS is supported as
  45. a target (as well as a source), either via JNDI or directly via JDBC.
  46. .. code:: xml
  47. <bean class="org.geoserver.taskmanager.external.impl.PostgisDbSourceImpl">
  48. <property name="name" value="mypostgisdb"/>
  49. <property name="host" value="hostname" />
  50. <property name="db" value="dbname" />
  51. <!-- optional --> <property name="schema" value="schema" />
  52. <property name="username" value="username" />
  53. <property name="password" value="password" />
  54. <!-- optional, for security purposes -->
  55. <property name="roles">
  56. <list>
  57. <value>ROLE1</value>
  58. <value>ROLE2</value>
  59. </list>
  60. </property>
  61. </bean>
  62. .. code:: xml
  63. <bean class="org.geoserver.taskmanager.external.impl.PostgisJndiDbSourceImpl">
  64. <property name="name" value="mypostgisjndidb" />
  65. <property name="jndiName" value="java:/comp/env/jdbc/my-jndi-source" />
  66. <!-- optional --> <property name="schema" value="schema" />
  67. <!-- optional, if database has different jndi name on target geoserver servers -->
  68. <property name="targetJndiNames">
  69. <map>
  70. <entry key="mygs" value="java:/comp/env/jdbc/my-jndi-source-on-mygs" />
  71. </map>
  72. </property>
  73. <!-- optional, for security purposes -->
  74. <property name="roles">
  75. <list>
  76. <value>ROLE1</value>
  77. <value>ROLE2</value>
  78. </list>
  79. </property>
  80. </bean>
  81. Roles can be specified for `security <#security>`__ purposes.
  82. Other database systems should generally work as a source database (not for publishing)
  83. using the GenericDbSourceImpl (this has been tested with MS SQL).
  84. .. code:: xml
  85. <bean class="org.geoserver.taskmanager.external.impl.GenericDbSourceImpl">
  86. <property name="name" value="mysqldb" />
  87. <property name="driver" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
  88. <property name="connectionUrl" value="jdbc:sqlserver://mysqldbhost:1433;database=mydb" />
  89. <property name="username" value="username" />
  90. <property name="password" value="password" />
  91. <property name="schema" value="dbo" />
  92. </bean>
  93. There is also specific support for Informix as a source database (not for publishing).
  94. .. code:: xml
  95. <bean class="org.geoserver.taskmanager.external.impl.InformixDbSourceImpl">
  96. <property name="name" value="myinformixdb" />
  97. <property name="driver" value="com.informix.jdbc.IfxDriver"/>
  98. <property name="connectionUrl" value="jdbc:informix-sqli://informix-server:1539" />
  99. <property name="username" value="username" />
  100. <property name="password" value="password" />
  101. </bean>
  102. It is also possible to use a source that does not support geometries, and translate them
  103. automatically from some raw type. To do this, one must create a table in the database
  104. that contains a list of all geometry columns that need to be translated. This can be
  105. configured as follows:
  106. .. code:: xml
  107. <bean name="geomtable" class="org.geoserver.taskmanager.external.impl.GeometryTableImpl">
  108. <!-- the name of your metadata table -->
  109. <property name="nameTable" value="Metadata_Geo" />
  110. <!-- the attribute name that contains table name -->
  111. <property name="attributeNameTable" value="table_name" />
  112. <!-- the attribute name that contains column name -->
  113. <property name="attributeNameGeometry" value="column_name" />
  114. <!-- the attribute name that contains geometry type -->
  115. <property name="attributeNameType" value="geometry_type" />
  116. <!-- the attribute name that contains SRID code -->
  117. <property name="attributeNameSrid" value="srid" />
  118. <!-- the type of conversion: WKT (string to geometry), WKB (binary to geometry), WKB_HEX (hex string to geometry) -->
  119. <property name="type" value="WKB_HEX" />
  120. </bean>
  121. <bean class="org.geoserver.taskmanager.external.impl.GenericDbSourceImpl">
  122. ....
  123. <property name="rawGeometryTable" ref="geomtable"/>
  124. </bean>
  125. .. _taskmanager_user_external_geoserver:
  126. External GeoServers
  127. ~~~~~~~~~~~~~~~~~~~
  128. Task Manager allows any number of external geoservers to be used as
  129. targets for layer publications. These are configured via the Spring
  130. configuration file.
  131. .. code:: xml
  132. <bean class="org.geoserver.taskmanager.external.impl.ExternalGSImpl">
  133. <property name="name" value="mygs"/>
  134. <property name="url" value="http://my.geoserver/geoserver" />
  135. <property name="username" value="admin" />
  136. <property name="password" value="geoserver" />
  137. <property name="supportMetadata" value="true" />
  138. </bean>
  139. The ''supportsMetadata'' field indicates whether this target geoserver contains the
  140. the :ref:`Metadata Community Module <community_metadata>`, which provides additional
  141. support for it in certain tasks.
  142. The configuration above will log-in to geoserver using basic authentication.
  143. Task Manager also supports geoservers protected with keycloak:
  144. .. code:: xml
  145. <bean class="org.geoserver.taskmanager.external.impl.ExternalKeycloakGSImpl">
  146. <property name="name" value="keycloakgs"/>
  147. <property name="url" value="http://my.geoserver/geoserver"/>
  148. <property name="username" value="keycloak_admin"/>
  149. <property name="password" value="keycloak_password"/>
  150. <property name="clientId" value="my clientid"/>
  151. <property name="clientSecret" value="my clientsecret"/>
  152. <property name="real" value="my realm"/>
  153. <property name="authUrl" value="http://my.keycloak.server/auth"/>
  154. <property name="supportMetadata" value="true" />
  155. </bean>
  156. .. _taskmanager_user_file_services:
  157. File Services
  158. ~~~~~~~~~~~~~
  159. File Services are used to upload and access files such as raster layers or vector files.
  160. They are configured via the Spring configuration file.
  161. Regular File Service
  162. ^^^^^^^^^^^^^^^^^^^^
  163. Regular file services provide support for rasters and vector files that are stored on the
  164. hard drive.
  165. .. code:: xml
  166. <bean class="org.geoserver.taskmanager.external.impl.FileServiceImpl">
  167. <property name="rootFolder" value="/tmp"/>
  168. <property name="name" value="Temporary Directory"/>
  169. <property name="roles">
  170. <list>
  171. <value>ROLE1</value>
  172. <value>ROLE2</value>
  173. </list>
  174. </property>
  175. </bean>
  176. Roles can be specified for `security <#security>`__ purposes.
  177. Non-absolute paths as rootFolder will be relative to the GeoServer Data Directory.
  178. Alternatively, it is also possible to use ``ResourceFileServiceImpl`` (same properties). This one only accepts relative paths and will use the data directory via the geoserver resource store, so that alternative implementations such as :ref:`JDBC Store <community_jdbcstore>` can be used. This might be useful for :ref:`Application Schemas <app-schema>`, for example.
  179. S3 File Service
  180. ^^^^^^^^^^^^^^^
  181. S3 File Services provide support for rasters that are stored on an S3
  182. compatible server.
  183. They do not need to be configured via the application context, but are
  184. taken from the properties file provided via the property
  185. ``s3.properties.location`` (see `S3
  186. DataStore <https://github.com/geotools/geotools/tree/main/modules/unsupported/s3-geotiff#geotiffs-hosted-on-other-amazon-s3-compatible-services>`__).
  187. A service will be created for each service and each bucket. We must add
  188. one line per alias to the ``s3.properties`` file:
  189. ``alias.s3.rootfolder=comma,separated,list,of,buckets``
  190. The above example will create five s3 file services: alias-comma,
  191. alias-separated, alias-list, alias-of and alias-buckets.
  192. Roles can optionally be specified for `security <#security>`__ purposes as follows:
  193. ``alias.bucket.s3.roles=comma,separated,list,of,roles``
  194. AWS File Service
  195. ^^^^^^^^^^^^^^^^
  196. Amazon AWS S3 buckets are also supported.
  197. .. code:: xml
  198. <bean class="org.geoserver.taskmanager.external.impl.AWSFileServiceImpl">
  199. <property name="rootFolder" value="/tmp"/>
  200. <property name="anonymous" value="false"/>
  201. <property name="awsRegion" value="us-west-1"/>
  202. <property name="roles">
  203. <list>
  204. <value>ROLE1</value>
  205. <value>ROLE2</value>
  206. </list>
  207. </property>
  208. </bean>
  209. Unless anonymous is set to true, the `default AWS client credential chain <https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#using-the-default-credential-provider-chain>`__ is used.
  210. Prepare script
  211. ^^^^^^^^^^^^^^^
  212. The task manager GUI allows immediate upload of files
  213. to file services for local publication.
  214. It may be handy to perform some preprocessing tasks
  215. on the uploaded data before publication (such as GDAL commands).
  216. You may do this by creating a file in the taskmanager configuration
  217. directory named `prepare.sh`. If the user ticks the prepare checkbox
  218. in the upload dialog, this script will be run with the uploaded file
  219. as its first parameter.
  220. Security
  221. --------
  222. Each configuration and each independent batch is associated with a
  223. workspace in GeoServer (when the workspace field is empty, it is
  224. automatically associated with the default workspace in geoserver). The
  225. configuration or batch takes its security permissions directly from this
  226. workspace.
  227. - If the user has reading permissions on the workspace, they may view
  228. the configuration or batch.
  229. - If the user has writing permissions on the workspace, they may run
  230. the batch or the batches in the configuration.
  231. - If the user has administrative permissions on the workspace, they may
  232. edit the configuration/batch.
  233. Each Database or File Service may be associated with a list of roles.
  234. If you do so, only users with those roles will have access to the database
  235. or file service in question. If you want to disable security restrictions,
  236. do not include the ``roles`` property at all (because an empty list will result
  237. in no access.)
  238. Graphical User Interface
  239. ------------------------
  240. Currently GeoServer Task Manager can only be configured and operated
  241. from the GeoServer WebGUI.
  242. Templates
  243. ~~~~~~~~~
  244. From the templates page, new templates can be created (or copied from
  245. existing templates), existing templates can be edited and removed.
  246. .. figure:: img/templates.png
  247. :alt: templates
  248. templates
  249. Once you open a new or existing template, attributes, tasks and batches
  250. can be edited. The attribute table adjusts automatically based on the
  251. information in the tasks table; and only the values must be filled in.
  252. In the task table, the name and parameters of each task can be edited,
  253. and new tasks can be created. Batches can be created and edited from
  254. here as well, however the template must exist in order to be able to do
  255. that (in case of a new template, you must click ``apply`` once before
  256. you can create new batches). New tasks must also be saved (again, via
  257. the ``apply`` button) before they can be added to a batch.
  258. .. figure:: img/template-db-workflow.png
  259. :alt: template db workflow
  260. template db workflow
  261. Configurations
  262. ~~~~~~~~~~~~~~
  263. From the `configurations <basic.html#configurations>`__ page, new
  264. configurations can be created from scratch or from templates (or copied
  265. from existing configurations), existing configurations can be edited and
  266. removed.
  267. .. figure:: img/configurations.png
  268. :alt: configurations
  269. configurations
  270. When removing a configuration, you have to option to do a *clean-up*,
  271. which will attempt to remove all resources (database tables, files,
  272. layers) that were created by (tasks of) this configuration. If this
  273. (partially) fails, the configuration will still be removed and the user
  274. will be notified.
  275. Once you open a new or existing configuration, attributes, tasks and
  276. batches can be edited.
  277. .. figure:: img/template-db-workflow-config2.png
  278. :alt: workflow config 2
  279. workflow config 2
  280. The attribute table adjusts automatically based on the information in
  281. the tasks table; and only the values must be filled in. In the task
  282. table, the name and parameters of each task can be edited, and new tasks
  283. can be created. Tasks can only be removed if they are not part of a
  284. batch any longer. Batches can only be removed if they are not running
  285. anywhere. When removing a task, you have to option to do a *clean-up*,
  286. which will attempt to remove all resources (database tables, files,
  287. layers) that were created by this task. If this (partially) fails, the
  288. task will still be removed and the user will be notified.
  289. Batches can be created and edited from here as well, however the
  290. configuration must exist in order to be able to do that (in case of a
  291. new configuration, you must click ``apply`` once before you can create
  292. new batches). New tasks must also be saved (again, via the ``apply``
  293. button) before they can be added to a batch. In case that the
  294. `conditions <basic.html#batches>`__ are met, batch runs can be started,
  295. and the status/history of current and past batch runs can be displayed.
  296. Current batch runs can be interrupted (which is not guaranteed to happen
  297. immediately).
  298. Import/Export
  299. ^^^^^^^^^^^^^
  300. It is also possible to import/export entire configurations to XML, for
  301. example to transfer them from one geoserver to another. The import button is
  302. on the configurations page, while the export button is on the page of a
  303. specific configuration. The user is responsible for making sure that the
  304. configuration is compatible with the other geoserver (available task extensions,
  305. attribute values,...).
  306. Batches
  307. ~~~~~~~
  308. From the `batches <basic.html#batches>`__ page, new independent batches
  309. (not associated with a configuration) can be created, existing batches
  310. can be edited and removed. All existing batches - independent as well as
  311. belonging to a configuration - are shown, unless they are special (if
  312. they start with a ``@``) or if the configuration has not yet been
  313. completed (see `initializing
  314. templates <basic.html#initializing-templates>`__).
  315. .. figure:: img/batches.png
  316. :alt: batches
  317. batches
  318. In case that the `conditions <basic.html#batches>`__ are met, batch runs
  319. can be started, and the status/history of current and past batch runs
  320. can be displayed. Current batch runs can be interrupted (which is not
  321. guaranteed to happen immediately).
  322. .. figure:: img/batchruns.png
  323. :alt: batchruns
  324. batchruns
  325. .. figure:: img/batchrun.png
  326. :alt: batchrun
  327. batchrun
  328. Once you open a new or existing batch, one can add or remove tasks from
  329. it and change the order of the tasks. You can also enable/disable the
  330. batch (if disabled, the batch is not scheduled) and choose the
  331. scheduling time. The user can choose between a daily schedule (with
  332. time), weekly (with day of week and time), monthly (with day of month
  333. and time) or specify a custom `cron
  334. expression <http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/crontrigger.html>`__.
  335. .. figure:: img/template-db-workflow-batch-sync.png
  336. :alt: batch synchronize
  337. batch synchronize
  338. Task Types
  339. ----------
  340. - ``CopyTableTask`` Copy a database table from one database to another.
  341. The user can specify a source database, source table name, target
  342. database and target table name. The source table name may also be a view.
  343. If the source does not contain a primary key column (f.e. if it is a view),
  344. an additional column 'generated_id', with an automatically generated primary key
  345. will be added to the destination table. The task will also copy all existing indexes.
  346. If the source table contains a geometry column but not a spatial index (f.e. if it is a view),
  347. a spatial index will automatically be added to the destination table.
  348. Supports commit/rollback by creating
  349. a temporary table.
  350. - ``CreateViewTask`` Create a view based on a single table. The user
  351. can specify the database, the table name, the selected fields and
  352. (optionally) a where condition. Supports commit/rollback by creating
  353. a temporary view.
  354. - ``CreateComplexViewTask`` Create a view based on a multiple tables.
  355. The user can specify the database and a whole query, where it can use
  356. any other configuration attribute in the form of '${placeholder}'.
  357. Supports commit/rollback by creating a temporary view.
  358. - ``CopyFileTask`` Copy a file from one file service to another.
  359. Commit/rollback is supported by a versioning system, where the
  360. version of the file is inserted into the file name. The location of
  361. the version number is specified in the path as ``###`` (or set
  362. auto-versioned to ``true`` to add the placeholder automatically
  363. before the extension dot). On commit,
  364. the older version is removed. On rollback, the newer version is
  365. removed. The publication tasks will automatically publish the latest
  366. version.
  367. - ``LocalDbPublicationTask`` Publish a database layer locally. The user
  368. can specify database, table and a layer name. Supports
  369. commit/rollback by advertising or removing the layer it created.
  370. - ``RemoteDbPublicationTask`` Publish a database layer to another
  371. geoserver. The user can specify a target geoserver, a source layer
  372. and a target database. All information is taken from the source layer
  373. except for the target database which may be different. Supports
  374. commit/rollback through creating a temporary (unadvertised) layer.
  375. This task also supports the version place holder or auto-versioning,
  376. in order to combine with the ``CopyFileTask``.
  377. - ``LocalFilePublicationTask`` Publish a file layer locally (raster or
  378. shapefile). The user can specify a file service, a file (which can be
  379. uploaded unto the service) and a layer name. Supports commit/rollback
  380. by advertising or removing the layer it created.
  381. - ``RemoteFilePublicationTask`` Publish a file layer locally (taster or
  382. shapefile). The user can specify a target geoserver, a source layer
  383. and a target file service and path (optional). All information is
  384. taken from the source layer except for the file service and path
  385. which may be different. Supports commit/rollback through creating a
  386. temporary (unadvertised) layer.
  387. - ``MetaDataSyncTask`` Synchronise the metadata between a local layer
  388. and a layer on another geoserver (without re-publishing). The user
  389. can specify a target geoserver, a local and a remote layer. Does not
  390. support commit/rollback. If the target geoserver supports the :ref:`Metadata Community Module <community_metadata>`
  391. native metadata attributes mapped to custom metadata attributes will be updated.
  392. Note that all of the publication tasks will synchronize metadata in the same
  393. way.
  394. - ``ConfigureCachedLayer`` Configure caching for a layer on a remote
  395. geoserver with internal GWC, synchronise the settings with the local
  396. geoserver. This task may turn caching on or off depending on local
  397. configuration.
  398. - ``ClearCachedLayer`` Clear (truncate) all tiles of a cached layer on
  399. a remote geoserver with internal GWC.
  400. - ``LocalAppSchemaPublicationTask`` Publish an :ref:`Application Schema <app-schema>` layer locally.
  401. This is exactly the same as ``LocalFilePublicationTask`` with the Application Schema mapping file
  402. as the file being published, and two additional features.
  403. * The mapping file may be provided as a template, with placeholders in the form of ``${placeholder}``.
  404. The placeholders are replaced by the values of the connection parameters of the database
  405. that is provided as parameter to the task. This makes it possible to fill in the underlying source
  406. database for different geoservers.
  407. For example: specify ``${jndiReferenceName}`` as source database connection parameter in the mapping file.
  408. * Multiple mapping files may be provided for a single layer (when the layer mapping uses included types), in the
  409. form of a ZIP file. The main mapping file and the ZIP file must have the same name before the extension.
  410. - ``RemoteAppSchemaPublicationTask`` Publish an :ref:`Application Schema <app-schema>` layer remotely.
  411. This is exactly the same as ``LocalFilePublicationTask`` with the Application Schema mapping file
  412. as the file being published, and two additional features:
  413. * The mapping file may be provided as a template, with placeholders in the form of ``${placeholder}``.
  414. The placeholders are replaced by the values of the connection parameters of the database
  415. that is provided as parameter to the task. This makes it possible to fill in the underlying source
  416. database for different geoservers.
  417. For example: specify ``${jndiReferenceName}`` as source database connection parameter in the mapping file.
  418. * Multiple mapping files may be provided for a single layer (when the layer mapping uses included types), in the
  419. form of a ZIP file. The main mapping file and the ZIP file must have the same name before the extension.
  420. - ``LayerSecuritySync`` this task will synchronise all :ref:`data access security rules <security_layer>` associated with a layer to the external geoserver. Warning: the task assumes that the same roles exist on both geoservers. Does not support commit/rollback.
  421. - ``WorkspaceSecuritySync`` this task will synchronise all :ref:`data access security rules <security_layer>` associated with a workspace to the external geoserver. Warning: the task assumes that the same roles exist on both geoservers. Does not support commit/rollback.
  422. - ``TimeStamp`` update a time stamp in a layer's metadata that represents
  423. the last time a layer's data has been updated. Since the data timestamp
  424. is part of the metadata, a metadata timestamp can also be updated.
  425. The task must be configured through its Spring Bean properties
  426. ``timeStampTaskType.dataTimestampProperty`` and
  427. ``timeStampTaskType.metadataTimestampProperty`` which represent the key (or
  428. key path) in the layer's resource metadata. If you are using the :ref:`Metadata Community Module <community_metadata>`
  429. you should set ``timeStampTaskType.metadataTimestampProperty=custom._timestamp``.
  430. - ``MetadataTemplateSync`` this task requires the :ref:`Metadata Community Module <community_metadata>`
  431. and the ``taskmanager-metadata`` submodule. It will synchronize all metadata linked to a specific metadata template.
  432. Useful when you change the template.
  433. Bulk Operations
  434. ---------------
  435. The task manager provides a number of bulk operation tools via an additional page in the GUI.
  436. The import tool is also available via a REST service.
  437. Run Batches
  438. ~~~~~~~~~~~
  439. A whole series of batches may be scheduled all at once.
  440. You specify a workspace, configuration name and batch name pattern to select the series of batches you want to schedule.
  441. You may specify how long to wait before starting to execute the batches.
  442. You may specify how long to wait in between execution of each batch. This option is strongly recommended not to overload your software and cause failures.
  443. .. figure:: img/bulk_runbatches.png
  444. Import Configurations
  445. ~~~~~~~~~~~~~~~~~~~~~
  446. The import tool allows bulk creation of an unlimited amount of
  447. configurations on the basis of a template and a CSV file with attribute
  448. values. Contrary to the rest of the configuration, this function is only
  449. exposed via a REST service and not via the GUI. The import tool will
  450. generate a new configuration for each line in the CSV file, except for
  451. the first. The first line must specify the attribute names which should
  452. all match attributes that exist in the template, plus ``name`` (required),
  453. ``description`` (optional) and ``workspace`` (optional) for the configuration
  454. metadata. The CSV file mustspecify a valid attribute value for each
  455. required attribute.
  456. Optionally, you may skip validation (at your own risk).
  457. As an alternative to using the GUI page, you may ``POST`` your CSV file to
  458. ``http://{geoserver-host}/geoserver/taskmanager-import/{template}[validate=false]``
  459. .. figure:: img/bulk_import.png
  460. Initialize Configurations
  461. ~~~~~~~~~~~~~~~~~~~~~~~~~
  462. If you have imported configurations in bulk based on an Initializing template, you
  463. may also want to initialize them in bulk. This works similarly to running batches in bulk.
  464. The configurations will be validated after initialization.
  465. .. figure:: img/bulk_initialize.png
  466. Examples
  467. --------
  468. Consider the following setup.
  469. Three geoservers:
  470. - ``work geoserver``: a geoserver only available in the local network,
  471. only used by administrators. New and updated data is published here
  472. as layers for the first time, to test both the validity of data and
  473. the publication configuration.
  474. - ``internal geoserver``: a geoserver only available in the local
  475. network, for internal users.
  476. - ``public geoserver``: a geoserver available on the internet, for the
  477. general public.
  478. Several databases:
  479. - ``multiple source databases``: these are databases provided by
  480. partners that provide new and updated data. they are not used to
  481. directly publish on a geoserver.
  482. - ``work database``: database used by the ``work geoserver`` where its
  483. vector data is stored.
  484. - ``internal database``: database used by the ``internal geoserver``
  485. where its vector data is stored.
  486. - ``public database``: database used by the ``public geoserver`` where
  487. its vector data is stored.
  488. A typical workflow for a new layer goes as follows:
  489. 1. A new table is copied from a ``source database`` to the
  490. ``work database`` and then published on the ``work geoserver``
  491. 2. After testing, the table is either copied to the
  492. ``internal database`` and published on the ``internal geoserver`` or
  493. copied to the ``public database`` and published on the
  494. ``public geoserver``.
  495. 3. Every week, data is synchronised between the three databases and
  496. metadata is synchronised between the two geoservers.
  497. Taskmanager should be installed only on the ``work geoserver``. Then we
  498. could make the following template:
  499. .. figure:: img/template-db-workflow.png
  500. :alt: template db workflow
  501. template db workflow
  502. with the following batches:
  503. .. figure:: img/template-db-workflow-batches.png
  504. :alt: template db workflow batches
  505. template db workflow batches
  506. The ``@Initialize`` batch:
  507. .. figure:: img/template-db-workflow-batch-init.png
  508. :alt: batch initialize
  509. batch initialize
  510. The ``PublishRemotely`` batch:
  511. .. figure:: img/template-db-workflow-batch-pubrem.png
  512. :alt: batch publish remotely
  513. batch publish remotely
  514. The ``Synchronize`` batch:
  515. .. figure:: img/template-db-workflow-batch-sync.png
  516. :alt: batch synchronize
  517. batch synchronize
  518. When we now create a new configuration based on this template we choose
  519. a source database, table name and layer name:
  520. .. figure:: img/template-db-workflow-config.png
  521. :alt: workflow config
  522. workflow config
  523. After clicking apply, the configuration is being initialized (the layer
  524. is created locally)...
  525. .. figure:: img/template-db-workflow-initializing.png
  526. :alt: initializing...
  527. initializing...
  528. We can now fill in the rest of the details, save, and make the remote
  529. publication. The synchronization is scheduled weekly.
  530. .. figure:: img/template-db-workflow-config2.png
  531. :alt: workflow config 2
  532. workflow config 2