cog.rst 9.7 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215
  1. .. _cog_plugin:
  2. COG (Cloud Optimized GeoTIFF) Support
  3. =====================================
  4. `COG <https://github.com/cogeotiff/cog-spec/blob/master/spec.md>`_ (Cloud Optimized GeoTIFF) is a regular GeoTIFF file, aimed at being hosted on a HTTP file server, whose internal organization is friendly for consumption by clients issuing `HTTP GET range <https://en.wikipedia.org/wiki/Byte_serving>`_ requests.
  5. The COG module allows to set configuration params to connect to a Cloud GeoTIFF, as well as adding JARs to the classpath needed to support that connection.
  6. Installation
  7. ------------
  8. As a community module, the package needs to be downloaded from the `nightly builds <https://build.geoserver.org/geoserver/>`_,
  9. picking the community folder of the corresponding GeoServer series (e.g. if working on the GeoServer main development branch nightly
  10. builds, pick the zip file form ``main/community-latest``).
  11. The module is provided as four different packages, depending on the connection type:
  12. * ``cog-http`` for HTTP connections without a particular cloud target
  13. * ``cog-aws`` for AWS S3 connections
  14. * ``cog-google`` for Google Cloud Storage connections
  15. * ``cog-azure`` for Azure Blob Storage connections
  16. To install the module, download the package you need, unpack the zip file contents into the GeoServer ``WEB-INF/lib`` directory and restart GeoServer.
  17. .. warning::
  18. The dependencies of the various cloud providers might interfere with each other, so please never install together the AWS, Google and Azure modules.
  19. COG GeoTIFF Configuration Panel
  20. -------------------------------
  21. The COG plugin does not add new stores, instead, it adds COG support to existing ones.
  22. When configuring a GeoTIFF store, a new checkbox is available: ``Cloud Optimized GeoTIFF (COG)``. Setting that will open a new section presenting the COG configuration parameters for this COG Store.
  23. .. figure:: images/cogparams.png
  24. COG Connection params
  25. Checking the ``Cloud Optimized GeoTIFF (COG)`` checkbox will provide new options:
  26. .. list-table::
  27. :widths: 20 80
  28. :header-rows: 1
  29. :stub-columns: 1
  30. * - Option
  31. - Description
  32. * - :guilabel:`URL`
  33. - (prefixed by ``cog://``) representing the connection URL to the COG Dataset.
  34. * - :guilabel:`Range Reader Settings`
  35. - Which type of Range Reader implementation. Values currently supported are HTTP, GoogleCloud, Azure, S3 the latter using an S3 Client
  36. * - :guilabel:`User Name / Access Key ID / Account Name`
  37. - Optional user name (HTTP) or Access Key ID (S3) or Account Name (Azure) in case the COG dataset requires authentication
  38. * - :guilabel:`Password / Secret Access Key / Account Key`
  39. - Password (HTTP) or Secret Access Key (S3) or Account Key (Azure) for the previous credential
  40. COG ImageMosaic Configuration
  41. -----------------------------
  42. Additional configuration parameters can be specified in the ImageMosaic indexer configuration, in order to properly configure a COG based ImageMosaic.
  43. :file:`indexer.properties`
  44. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  45. .. list-table::
  46. :widths: 15 5 80
  47. :header-rows: 1
  48. :stub-columns: 1
  49. * - Parameter
  50. - Mandatory?
  51. - Description
  52. * - Cog
  53. - Y
  54. - A boolean flag (true/false) to be set (Cog=true) in order to signal that the ImageMosaic is a COG data mosaic.
  55. * - CogRangeReader
  56. - N
  57. - Specifies the desired RangeReader implementation performing the Range Reads requests.
  58. * - CogUser
  59. - N
  60. - Credential to be set whenever basic HTTP authentication is needed to access the COG Datasets or an S3 Access KeyID is required or an Azure AccountName is required
  61. * - CogPassword
  62. - N
  63. - Password for the above user OR Secret Access Key for the above S3 KeyId or AccountKey for the above Azure AccountName.
  64. .. _cog_plugin_rangereader:
  65. COG RangeReader
  66. ```````````````
  67. The following table provides the values for the ``CogRangeReader`` based on the type of target storage:
  68. .. list-table::
  69. :widths: 20 80
  70. :header-rows: 1
  71. :stub-columns: 1
  72. * - Storage type
  73. - Class name
  74. * - HTTP
  75. - Can be omitted, or set to ``it.geosolutions.imageioimpl.plugins.cog.HttpRangeReader``
  76. * - AWS S3
  77. - ``it.geosolutions.imageioimpl.plugins.cog.S3RangeReader``
  78. * - Google Cloud
  79. - ``it.geosolutions.imageioimpl.plugins.cog.GSRangeReader``
  80. * - Azure
  81. - ``it.geosolutions.imageioimpl.plugins.cog.AzureRangeReader``
  82. COG Global Settings
  83. -------------------
  84. The GeoServer Global Settings page contains the default COG settings presented when setting up a new COG GeoTIFF Store.
  85. .. figure:: images/globalcogsettings.png
  86. Default Global COG Settings
  87. Image locations
  88. ---------------
  89. For images served by a HTTP server, a HTTP URL must be used.
  90. For images served by S3 or Google Cloud, it's possible to use both the public HTTP URL,
  91. or the idiomatic URIS, for example:
  92. * ``s3://landsat-pds/c1/L8/153/075/LC08_L1TP_153075_20190515_20190515_01_RT/LC08_L1TP_153075_20190515_20190515_01_RT_B2.TIF``
  93. * ``gs://gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_B11.TIF``
  94. HTTP Client (OkHttp) configuration
  95. ----------------------------------
  96. HTTP client configuration (based on `OkHttp client <https://square.github.io/okhttp/>`_) can be specified through Environment variables.
  97. .. list-table::
  98. :widths: 15 80
  99. :header-rows: 1
  100. :stub-columns: 1
  101. * - Environment Variable
  102. - Description
  103. * - `IIO_HTTP_MAX_REQUESTS`
  104. - The maximum number of requests to execute concurrently. Above this requests queue in memory, waiting for the running calls to complete. (Default 128)
  105. * - `IIO_HTTP_MAX_REQUESTS_PER_HOST`
  106. - The maximum number of requests for each host to execute concurrently. (Default 5)
  107. * - `IIO_HTTP_MAX_IDLE_CONNECTIONS`
  108. - The maximum number of idle connections. (Default 5)
  109. * - `IIO_HTTP_KEEP_ALIVE_TIME`
  110. - The Keep alive time (in seconds), representing maximum time that excess idle threads will wait for new tasks before terminating. (Default 60)
  111. AWS S3 Client configuration
  112. ---------------------------
  113. A single S3 Asynchronous Client will be used for the same region and alias (url schema, i.e. http, https).
  114. The following Environment Variables can be set to customize the pool for the asynchronous client for that particular alias.
  115. On the table below, replace the "$ALIAS$" template with HTTP or HTTPS or S3 if you are configuring properties for these schema.
  116. .. list-table::
  117. :widths: 15 80
  118. :header-rows: 1
  119. :stub-columns: 1
  120. * - Environment Variable
  121. - Description
  122. * - `IIO_$ALIAS$_AWS_CORE_POOL_SIZE`
  123. - The core pool size for the S3 Client (Default 50)
  124. * - `IIO_$ALIAS$_AWS_MAX_POOL_SIZE`
  125. - The maximum number of thread to allow in the pool for the S3 Client (Default 128)
  126. * - `IIO_$ALIAS$_AWS_KEEP_ALIVE_TIME`
  127. - The Keep alive time (in seconds), representing maximum time that excess idle threads will wait for new tasks before terminating. (Default 10)
  128. * - `IIO_$ALIAS$_AWS_USER`
  129. - Default user (access key ID) for AWS basic authentication credentials
  130. * - `IIO_$ALIAS$_AWS_PASSWORD`
  131. - Default password (secret access key) for AWS basic authentication credentials
  132. * - `IIO_$ALIAS$_AWS_REGION`
  133. - Default AWS region
  134. * - `IIO_$ALIAS$_AWS_ENDPOINT`
  135. - Endpoint to Amazon service or any other S3-compatible service run by a third-party
  136. Google Cloud storage configuration
  137. ----------------------------------
  138. The credentials to access Google Cloud cannot be provided as username and password (an authentication
  139. method that Google Cloud does not support), but need to be provided via a system variable pointing
  140. to the key file::
  141. set GOOGLE_APPLICATION_CREDENTIALS=/path/to/the/key-file.json
  142. export GOOGLE_APPLICATION_CREDENTIALS
  143. Azure configuration
  144. -------------------
  145. A single Azure Client will be used for the same container.
  146. Account and container will be retrieved from the provided Azure URL.
  147. The following System Properties can be set to customize client properties where missing.
  148. .. list-table::
  149. :widths: 15 80
  150. :header-rows: 1
  151. :stub-columns: 1
  152. * - System property
  153. - Description
  154. * - azure.reader.accountName
  155. - The Azure Account Name
  156. * - azure.reader.accountKey
  157. - The Azure Account Key for the above Account
  158. * - azure.reader.container
  159. - The default container for the above Account
  160. * - azure.reader.prefix
  161. - The optional prefix containing blobs
  162. * - azure.reader.maxConnections
  163. - The max number of connections supported by the underlying Azure client
  164. Client configuration (System Properties)
  165. ----------------------------------------
  166. Note that all the IIO settings reported in the previous tables can also be specified using System Properties instead of Environment variables.
  167. You just need to replace UPPER CASE words with lower case words and underscores with dots.
  168. So, the value for Maximum HTTP requests can be specified by setting either a ``IIO_HTTP_MAX_REQUESTS`` Environment variable or a ``iio.http.max.requests`` Java System Property alternatively (Environment variables are checked first).
  169. By default, when accessing a COG, an initial chunk of 16 KB is read in attempt to parse the header so that the reader will have the offset and length of the available tiles. When dealing with files hosting many tiles, it is possible that the whole header won't fit in the initial chunk. In this case additional reads (chunks of the same size) will be progressively made to complete loading the header.
  170. A ``it.geosolutions.cog.default.header.length`` system property can be configured to set the length (in bytes) of the reading chunk. Tuning this so that the header is read with few extra requests can help improve performance. A value too large can cause memory consumption issues and will reduce efficiency, as un-necessary data will be read.