• If you get an error about  --create-bucket-configuration 

• please change line 116 in file cloud-automation/gen3/bin/workon.sh  and replace this line with the following
  gen3_aws_run aws s3api create-bucket --acl private --bucket "$GEN3_S3_BUCKET" --region $(aws configure get $GEN3_PROFILE.region)

• If the error says there is an issue with the backend for S3, please do the following

• After the gen3 workon command, do gen3 cd and ls. You should be able to see a file named backend.tfvars
  

• Open that file and add the following line in the file
  profile = <>
  

• Once done, re-run the gen3 workon command

• Windows:

• Open Command Prompt or PowerShell:

• Press Win + R, type cmd or powershell, and hit Enter.

• Update kubectl:

• Run kubectl version --client to check the current version.
  

• If outdated, download the latest version from the Kubernetes website.
  

• Replace the old kubectl.exe with the new one in your PATH environment variable.

• Update AWS CLI:

• Run aws --version to check the current version.
  

• If outdated, download the latest version from the AWS website.
  

• Install it and update your PATH if necessary.

• macOS:

• Open Terminal:

• Use Spotlight (Cmd + Space) and search for "Terminal" to open it.

• Update kubectl:

• Run kubectl version --client to check the current version.
  

• Use Homebrew to update: brew upgrade kubectl or specify a version like brew install kubectl@.

• Update AWS CLI:

• Run aws --version to check the current version.
  

• Use Homebrew to update: brew upgrade awscli.

• Ubuntu (Linux):

• Open Terminal:

• Use Ctrl + Alt + T or search for "Terminal" in your applications.

• Update kubectl:

• Run kubectl version --client to check the current version.
  

• Use curl or wget to download the latest version from the Kubernetes GitHub releases.
  

• Install the downloaded binary and update your PATH if necessary.

• Update AWS CLI:

• Run aws --version to check the current version.
  

• Use pip to update AWS CLI: sudo pip install --upgrade awscli.

**

1. The first step for our helm installation of Gen3 is to prepare our deployment to authenticate appropriately with AWS Secrets Manager.

2. If you already have a basic helm installed on your computer, execute this script - external_secrets.sh from repository support-scripts-gen3 to install the external_values helm chart. This will allow us to use external secrets in helm charts. Below are the input variables required for running the script.

• <>: AWS account number in which the commons would spin up
  
• <>: AWS region in which EKS is spun up.
  
• <>: Name for an IAM Policy
  
• <>: Name of IAM user who would be using the above IAM policy to spin up helm charts, the name of secrets_user created in cloud-automation
  
• <>: AWS profile from the local/VM which can access the AWS Account number given, if the profile value is default
  

Note (To-do): We require the AWS Secret Key and Secret Access Key for the IAM user mentioned in this step. These credentials will be utilized in the next step.

2. Our infrastructure deployment from cloud-automation will have provisioned a “secrets_user” in IAM which is used in the previous step. We want to provide security credentials for this user ahead of time to enable our deployment. This will allow us to authenticate with secrets-manager without maintaining AWS creds in the values file.
   

kubectl create secret generic prelim-aws-config \  
    --from-literal=access-key=’<<aws-access-key-id>>’ \  
    --from-literal=secret-access-key=’<<aws-secret-access-key>>’  
Copy

2. The next step is to configure our Authentication service to utilize our desired Identity Provider (Google):

3. Visit and modify the fence-config-template.yaml

4. Replace <<your-website-URL>>

5. Replace <<your-google-client-id>>

6. Replace <<your-google-client-secret>>

7. Store the updated configuration in AWS Secrets Manager

8. Name the secret <<your-vpc-name>>-fence-config

1. The very first thing we must do is add or clone the Gen3 Helm repository. This is how we package up all the components that make up Gen3 and make them accessible to the public.

git clone [https://github.com/occ-data/gen3-helm  
](https://github.com/occ-data/gen3-helm)cd gen3-helm/helm
Copy

2. Once you have your repo added, you can install it with the command

helm upgrade --install dev ./gen3 -f /path/to/updated/values.yaml
Copy

3. Enable access to our deployments over HTTPS by setting up a load balancer that references a valid, signed certificate for your domain.  This can be done via two options:

4. (CDTS Official) Gen3 facilitates this process via cloud-automation with the following command:
   

  gen3 kube-setup-ingress  
Copy

This command will perform a helm installation of the alb-ingress-controller after pre-provisioning IAM roles/policies for the ingress controller, ACL rules for a WAF, etc.  
  
Your deployment should now be available over HTTPS.  
  
Note:  We can perform this with more standard tooling such as cert-manager and an ingress controller with a more robust solution/community behind it. (Explained in step b)  
  
  
Copy

2. (Alternative to step a)  Install an ingress controller (NGINX) to your cluster to provision an ALB.  Automate certificate signing requests by installing cert-manager and preparing issuers for your desired Certificate Authority.

3. Install NGINX.  This will provision an ALB resource in your AWS account
   

helm upgrade --install ingress-nginx ingress-nginx \  
      --repo https://kubernetes.github.io/ingress-nginx \  
      --namespace ingress-nginx \  
      --create-namespace  
Copy

2. Install cert-manager.  This will automate certificate signing requests
   

  helm repo add jetstack https://charts.jetstack.io \  
     --force-update  
Copy

  helm install   cert-manager jetstack/cert-manager \  
     --namespace cert-manager \  
     --create-namespace \  
     --version v1.14.4 \  
     --set installCRDs=true  
Copy

3. Create your issuers.  For our example, we will use LetsEncrypt

4. Staging Certificate (useful for debugging certificate signing requests)

kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: ClusterIssuer

metadata:

  name: letsencrypt-staging

spec:

  acme:

    email: email@example.com

    preferredChain: a    privateKeySecretRef:

      name: letsencrypt-staging

    server: https://acme-staging-v02.api.letsencrypt.org/directory

    solvers:

      - http01:

          ingress:

            class: nginx

EOF
Copy

2. Production Certificates (use this once your staging certificate has been successfully signed)

kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: ClusterIssuer

metadata:

  name: letsencrypt-prod

spec:

  acme:

    email: email@example.com

    preferredChain: ''

    privateKeySecretRef:

      name: letsencrypt-prod

    server: https://acme-v02.api.letsencrypt.org/directory

    solvers:

      - http01:

          ingress:

            class: nginx

EOF
Copy

3. Running these scripts will allow helm charts to update ingress configuration for revproxy via Helm chart values 

(Note: If you are following the example helm charts provided in the demonstration, The configuration below is already provided in the chart)

1. For example: in the YAML file for values, Values.revproxy.ingress has the class name nginx with annotations

2. cert-manager.io/cluster-issuer:"letsencrypt-prod"

This annotation specifies the cluster issuer which we previously deployed

2. acme.cert-manager.io/http01-edit-in-place: "true"

revproxy:

  enabled: true

  ingress:

    enabled: true

    className: "nginx"

    annotations:

      acme.cert-manager.io/http01-edit-in-place: "true"

      cert-manager.io/cluster-issuer: "letsencrypt-prod"

    hosts:

    - host: gen3-demo.cloud.krum.io

      paths:

        - path: /

          pathType: Prefix

    tls:

      - secretName: cm-gen3-certs-prod

        hosts:

          - gen3-demo.cloud.krum.io

4. Upon finishing this configuration, you can proceed to deploy the Gen3 Helm charts. We offer a simplified Helm chart with configurations for running a basic Gen3 version. You can find the helm chart in the helm chart - gen3_values.yaml within the support-scripts-gen3. There are some variables in the charts which are explained in the readme file here. 

Details about each microservice deployed in the sample Helm chart are provided after completing step 6. You can customize the Helm chart according to your specific requirements. For more information about available microservices and their configuration, refer to CONFIGURATION.md, which contains detailed instructions and a comprehensive range of configuration options for each service.

5. After setting up your Helm environment, either by following the provided example or using the values.yaml file in the gen3-helm directory on your system (located at demo-gen3/gen3-helm/helm/gen3/values.yaml for the Gen3 Helm code), you can execute the helm upgrade commands to initiate the Helm process.

Note: To deploy Gen3 Helm beyond this document, ensure that you have the necessary prerequisites installed and configured. The following section outlines a set of microservices and instructions for updating the chart to run a basic version of Gen3. For detailed configuration of any unlisted microservices, consult the configuration documentation for further insights. 

After making the required updates in the Helm charts or if you wish to apply overrides, you can achieve this by supplying the values.yaml file. (Note: Before executing this command, ensure you are in the demo-gen3/gen3-helm/helm/gen3/ folder. You can confirm your current directory using the pwd command.)

helm upgrade --install dev gen3/gen3 -f ./values.yaml  
Copy

If you want to provide overrides you can do so by passing in one, or several values.yaml files. For example, if you want to pass in user.yaml and fence-config (NB! New format, check out sample files in this folder)

helm upgrade --install dev gen3/gen3  -f values.yaml -f fence-config.yaml -f user.yaml
Copy

Note: Each time you modify the values.yaml file, you need to execute the helm upgrade command to ensure that the changes are applied to both the Helm charts and Kubernetes.

Before we start examining the pre-configured file in detail, let's take a look at the list of sub-gen3 helm charts for gen3 microservices that are utilized to set up a basic Gen3 data commons as outlined in this script.

• global (this is a helm chart variable, not a gen3 microservice)
  
• arborist
  
• aws-es-proxy
  
• etl
  
• fence
  
• guppy
  
• indexd
  
• peregrine
  
• portal
  
• revproxy
  
• sheepdog

Global configuration to execute helm charts is defined in the global section. This part includes variables needed to connect to various AWS resources like the Postgres database, AWS Certificate, Secrets Manager, etc. It also involves tasks such as fetching the data dictionary, defining GitOps processes, and hosting configurations.

The first variable in global is aws in which the secret for AWS credentials is given. The AWS credentials are stored in Kubernetes secret prelim-aws-config 

aws:

   enabled: true

   useLocalSecret:

     enabled: true

     localSecretName: prelim-aws-config  
Copy

Note: All database credentials are stored in AWS Secrets manager with the naming convention of  {{environment}}-{{service}}-creds, these secrets are then fetched and stored into the Kubernetes secrets because of the flag externalSecrets set to yes. As shown in the configuration below, postgres db and external secrets are enabled.

postgres:

  dbCreate: true

externalSecrets:

  deploy: true
Copy

The host configuration is also part of the global chart, 

• Dev: The value "yes" or "no" in the development environment setting determines whether Postgres and Elasticsearch should be deployed or not.
  
• Environment:  name of the VPC/environment
  
• Hostname: URL of gen3 data commons without http:// or https:// (point 5 in Variables asked)
  
• revproxyArn: ARN of the certificate for the domain/website, (point 7 in Variables asked)
  
• dictionaryUrl: S3 URL from which data dictionary will be fetched

dev: false

environment: <<your-vpc-name>>

hostname: <<your-website-URL>>

revproxyArn: arn:aws:acm:us-east-1:XXXXXXXXXX:certificate/XXXXXXXXX # <<your-website-aws-certificate-arn>>

dictionaryUrl: https://s3.amazonaws.com/XXXXXXXX/XXXXX/XXXXXXXX/XXXX.json  # <<your-data-dictionary-URL>>

portalApp: gitops

publicDataSets: true
Copy

Alongside these, the other values are also required for the global helm chart for gen3, the description of these values is provided here

tierAccessLevel: libre

tierAccessLimit: "1000"

netPolicy: true

dispatcherJobNum: 10

ddEnabled: false

manifestGlobalExtraValues:

  fence_url: https://<<your-website-URL>>/user
Copy

Arborist is a key part of the Gen3 stack, managing access control through policies, roles, and resources. It organizes resources hierarchically and grants permissions based on defined roles.  In the Gen3 stack, the arborist works closely with the fence, which handles user authentication and issues JWT tokens containing authorization policies. Other services can check user authorization by sending the JWT to an arborist, ensuring secure access control without storing user credentials. The arborist is an open-source microservice and can be located on its git code base here

In helm charts, the arborist helm chart can be deployed either as an entity or part of gen3 helm charts. The list of values for helm chart arborist can be found here

To use Arborist in Helm, the basic variables are to enable the service and give database credentials. Since we have database secrets stored in AWS Secrets Manager, the only thing needed in our version of the helm chart is to enable arborist.

arborist:

  enabled: true
Copy

aws-es-proxy is a small web server application sitting between your HTTP client (browser, curl, etc...) and Amazon Elasticsearch service. This service is responsible for connecting the elastic search to gen3. 

Aws-es-proxy can be deployed as an individual chart or as a part of the gen3 helm chart. In the example helm chart, aws-es-proxy would need to be enabled, and elastic search endpoint should be provided

aws-es-proxy:

  enabled: true

  esEndpoint: <>

 externalSecrets:

   awsCreds: <>-aws-es-proxy-creds

ETL helm sub chart in Gen3 helm contains etl configuration to tables and fields to ETL to ElasticSearch. The field mapping to get values from the graph model into elastic search with transformation. As shown below, the ETL helm chart contains esEndpoint to local cluster (this is localized due to aws-es-proxy) and etlMapping to show mappings to graph model and transformation of values (i.e gender and race)

etl:

  enabled: true

  esEndpoint: elasticsearch.default.svc.cluster.local

  etlMapping:

    mappings:

      - name: dev_case

        doc_type: case

        type: aggregator

        root: case

        props:

          - name: submitter_id

          - name: project_id

        flatten_props:

          - path: demographics

            props:

              - name: gender

                value_mappings:

                  - female: F

                  - male: M

              - name: race

                value_mappings:

                  - american indian or alaskan native: Indian

              - name: ethnicity

              - name: year_of_birth

        aggregated_props:

          - name: _samples_count

            path: samples

            fn: count
Copy

Fence microservice provides an authentication and authorization framework for all Gen3 services & resources. The fence separates protected resources from the outside world, allowing only trusted entities to enter. The fence is an open-source Gen3 microservice whose git code base can be found here

This document would only focus on deploying the basic configuration of the fence. The helm deployment configuration for the fence can be found in gen3-helm github. This link consists of the table with details about all the variables defined for further development.

In the basic configuration of the fence helm chart which is shown below, the image of the microservice is defined. The image needs to be updated periodically as microservices are constantly changing. Alongside the external secret defined for fence db with the naming convention of  {{environment}}-{{service}}-creds, and fence-config secret, Other externalSecrets are given such as 

• createK8sFenceConfigSecret: this will create the Helm fence-config secret
  
• createK8sJwtKeysSecret: this will create the Helm fence-jwt-keys secret
  
• createK8sGoogleAppSecrets: this will create the Helm fence-google-app-creds-secret and fence-google-storage-creds-secret secrets

fence:

  enabled: true

  image:

    tag: 2024.04

  externalSecrets:

  fenceConfig: <<your-vpc-name>>-fence-config

    createK8sFenceConfigSecret: false

    createK8sJwtKeysSecret: true

    createK8sGoogleAppSecrets: true

  USER_YAML: |
Copy

Guppy is a server that supports GraphQL queries on data from Elasticsearch. Guppy can be set up either as a standalone helm chart or as part of the gen3-helm setup. In the basic script we're showcasing here, the outline of the configuration details for Guppy is below. The esEndpoint for elastic search is given alongside the indices, make sure that the name of indices matches the name of mapping in etlMapping

guppy:

  enabled: true

  image:

    tag: 2024.04 

  esEndpoint: http://elasticsearch.default.svc.cluster.local:9200

  indices:

  - index: dev_case

    type: case
Copy

Indexd is a hash-based data indexing and tracking service providing 128-bit globally unique identifiers. It is designed to be accessed via a REST-like API or a client. It supports distributed resolution with a central resolver talking to other Indexd servers.

Indexd can be a standalone helm chart or part of Gen3 helm configuration. In the provided script, createK8sServiceCredsSecret is given to create the indexd-service-creds secret. This service also uses external secret defined for db with the naming convention of  {{environment}}-{{service}}-creds.

indexd:

  enabled: true

  image:

    tag: 2024.04 

  externalSecrets:

    createK8sServiceCredsSecret: true

  defaultPrefix: <<your-index-prefix>>

  
Copy

Peregrine provides a query interface to get insights into data in Gen3 Commons. It mostly works with the Postgres database to perform queries. This microservice can be implemented as a stand-alone helm chart or part of the Gen3 Helm chart. The code snippet below shows how peregrine is included in the helm chart used for demonstration today. This script has arboristUrl which allows the peregrine to talk to the arborist.

peregrine:

  enabled: true

  image:

    tag: 2024.04

  arboristUrl: http://arborist-service

The portal microservice is responsible for spinning up the UI front of the website and also supports some basic interaction with Gen3. You can customize the logo, the colors, what elements are to be shown in the UI, and many more using the gitops configuration in the portal microservice. The portal can be an individual helm chart or part of the Gen3 helm chart. The code snippet below shows the configuration for the portal. In gitops, you can give the base64 encoded code for the favicon image <> and logo <> for Gen3 Commons. The json portion consists of gitops json which can have the format described in the link here

portal:

  enabled: true

  image:

    tag: 2023.04   

  gitops:

    favicon: <<base-64-of-favicon-image>>

    logo: <<base-64-of-logo-image>>

    json: | 
Copy

Revproxy is a core service to a commons that handles networking within the Kubernetes cluster. Revproxy is an NGINX container managing cluster endpoints. Revproxy is a vital nginx container that stores cluster endpoint data, requiring a setup of endpoints to enable traffic routing and normal operation. Revproxy can be an independent helm deployment or part of the Gen3 helm chart as shown below with a snippet for the demonstration helm chart. 

The configuration here has nginx className, multiple annotations, hosts, and tls configurations to import the certificate from the AWS certificate manager to spin up load balancers (ALB- application load balancer) to run the Gen3 Commons

  acme.cert-manager.io/http01-edit-in-place: "true"

 cert-manager.io/cluster-issuer: "letsencrypt-prod"

revproxy:

  enabled: true

  ingress:

    enabled: true

    className: "nginx"

    annotations:

      acme.cert-manager.io/http01-edit-in-place: "true"

      cert-manager.io/cluster-issuer: "letsencrypt-prod"

    hosts:

    - host: <<your-website-URL>>

      paths:

        - path: /

          pathType: Prefix

    tls:

      - secretName: cm-gen3-certs-prod        # pragma: allowlist secret

        hosts:

          - <<your-website-URL>>
Copy

Sheepdog is a core service that handles data submission. Data gets submitted to the commons, using the dictionary as a schema, which is reflected within the sheepdog database. Sheepdog has the configuration to connect to fenceUrl to check user permissions for data submission, also it has volumeMounts for volumes to mount to the container which takes configurations from the wsgi.py file

  

sheepdog:

  enabled: true

  image:

    tag: 2024.04 

  fenceUrl: https://<<your-website-URL>>/user

  volumeMounts:

  - name: "config-volume"

    readOnly: true

    mountPath: "/var/www/sheepdog/settings.py"

    subPath: "wsgi.py"
Copy

**