Commit 4d7d096a authored by David Diederich's avatar David Diederich
Browse files

Initial Import

parents
# Eliminate pkg, bin
pkg/*
bin/*
# build scripts
build-scripts\publish-swagger.sh
# Binaries for programs and plugins
*.exe
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
cloc/
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# vendor folder
src/entitlements/vendor/
src/github.com/
src/golang.org/
# Project-local glide cache, RE: https://github.com/Masterminds/glide/issues/736
.glide/
.idea/
*.iml
# Files required for running locally
src/entitlements/entitlements-bucket-access.json
src/entitlements/*-services.json
bucket/
src/entitlements/swagger-ui
# Ignoring .DS_Store
.DS_Store
# local docker build
.cover/
cobertura/
testresults/
.swagger-codegen/
\ No newline at end of file
# Naming
Names matter
Programs are full of names. Names have costs and benefits.
**Costs: space and time:**
Names need to be in short term memory when reading code.
You can only fit so many. Longer names take up more space.
**Benefits: information:**
A good name is not only a referent, it conveys information.
__Use the shortest name that carries the right amount of information in its context.__
##Name Style
1. Use camelCase, not_underscores.
Local variable names should be short, typically one or two characters.
2. Package names are usually one lowercase word.
3. Global variables should have longer names.
##Doc comments
Doc comments precede the declaration of an exported identifier:
```
// Join concatenates the elements of elem to create a single string.
// The separator string sep is placed between elements in the resulting string.
func Join(elem []string, sep string) string {
```
###Writing doc comments
Doc comments should be English sentences and paragraphs.
They use no special formatting beyond indentation for preformatted text.
Doc comments should begin with the noun they describe.
```
// Join concatenates… good
// This function… bad
```
Package docs go above the package declaration:
```
// Package fmt
package fmt
```
##References
1. https://godoc.org/code.google.com/p/go.tools/cmd/vet
2. https://talks.golang.org/2014
FROM golang:1.13 AS builder
WORKDIR /go/src/app/src/entitlements
RUN go get -u github.com/golang/dep/cmd/dep
COPY . /go/src/app/
ENV GOPATH=/go/src/app
RUN dep ensure
RUN CGO_ENABLED=0 GOOS=linux go build -a -o app_entl .
RUN go test ./...
FROM alpine:latest
RUN apk --no-cache add ca-certificates
RUN apk add curl
WORKDIR /root/
COPY --from=builder /go/src/app/src/entitlements/app_entl .
COPY --from=builder /go/src/app/src/entitlements/config config
CMD ["./app_entl"]
EXPOSE 8080
## Delegate domain-wide authority to service account
Follow the steps as outlined in the link below to setup the service account:
https://developers.google.com/admin-sdk/directory/v1/guides/delegation
The following scopes are required for service account:
```
View and manage the provisioning of groups on your domain https://www.googleapis.com/auth/admin.directory.group
View and manage group subscriptions on your domain https://www.googleapis.com/auth/admin.directory.group.member
View group subscriptions on your domain https://www.googleapis.com/auth/admin.directory.group.member.readonly
View groups on your domain https://www.googleapis.com/auth/admin.directory.group.readonly
View and manage the provisioning of users on your domain https://www.googleapis.com/auth/admin.directory.user
```
\ No newline at end of file
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
name = "cloud.google.com/go"
packages = ["compute/metadata","iam","internal","internal/optional","internal/version","storage"]
revision = "767c40d6a2e058483c25fa193e963a22da17236d"
version = "v0.18.0"
[[projects]]
name = "github.com/davecgh/go-spew"
packages = ["spew"]
revision = "346938d642f2ec3594ed81d874461961cd0faa76"
version = "v1.1.0"
[[projects]]
name = "github.com/dgrijalva/jwt-go"
packages = ["."]
revision = "dbeaa9332f19a944acb5736b4456cfcc02140e29"
version = "v3.1.0"
[[projects]]
name = "github.com/franela/goblin"
packages = ["."]
revision = "74c9fe110d4bfd04c222a089a309e0a97e258534"
version = "0.0.1"
[[projects]]
name = "github.com/golang/protobuf"
packages = ["proto","protoc-gen-go/descriptor","ptypes","ptypes/any","ptypes/duration","ptypes/timestamp"]
revision = "925541529c1fa6821df4e44ce2723319eb2be768"
version = "v1.0.0"
[[projects]]
name = "github.com/googleapis/gax-go"
packages = ["."]
revision = "317e0006254c44a0ac427cc52a0e083ff0b9622f"
version = "v2.0.0"
[[projects]]
name = "github.com/gorilla/context"
packages = ["."]
revision = "1ea25387ff6f684839d82767c1733ff4d4d15d0a"
version = "v1.1"
[[projects]]
name = "github.com/gorilla/mux"
packages = ["."]
revision = "53c1911da2b537f792e7cafcb446b05ffe33b996"
version = "v1.6.1"
[[projects]]
name = "github.com/pmezard/go-difflib"
packages = ["difflib"]
revision = "792786c7400a136282c1664665ae0a8db921c6c2"
version = "v1.0.0"
[[projects]]
name = "github.com/rs/cors"
packages = ["."]
revision = "7af7a1e09ba336d2ea14b1ce73bf693c6837dbf6"
version = "v1.2"
[[projects]]
name = "github.com/stretchr/testify"
packages = ["assert"]
revision = "12b6f73e6084dad08a7c6e575284b177ecafbc71"
version = "v1.2.1"
[[projects]]
branch = "master"
name = "golang.org/x/net"
packages = ["context","context/ctxhttp","http2","http2/hpack","idna","internal/timeseries","lex/httplex","trace"]
revision = "cbe0f9307d0156177f9dd5dc85da1a31abc5f2fb"
[[projects]]
branch = "master"
name = "golang.org/x/oauth2"
packages = [".","google","internal","jws","jwt"]
revision = "543e37812f10c46c622c9575afd7ad22f22a12ba"
[[projects]]
branch = "master"
name = "golang.org/x/text"
packages = ["collate","collate/build","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
revision = "4e4a3210bb54bb31f6ab2cdca2edcc0b50c420c1"
[[projects]]
branch = "master"
name = "google.golang.org/api"
packages = ["admin/directory/v1","gensupport","googleapi","googleapi/internal/uritemplates","googleapi/transport","internal","iterator","option","storage/v1","transport/http"]
revision = "c76a25da6c114df45a86acb8873ae83eda1430b9"
[[projects]]
name = "google.golang.org/appengine"
packages = [".","internal","internal/app_identity","internal/base","internal/datastore","internal/log","internal/modules","internal/remote_api","internal/urlfetch","urlfetch"]
revision = "150dc57a1b433e64154302bdc40b6bb8aefa313a"
version = "v1.0.0"
[[projects]]
branch = "master"
name = "google.golang.org/genproto"
packages = ["googleapis/api/annotations","googleapis/iam/v1","googleapis/rpc/status"]
revision = "2b5a72b8730b0b16380010cfe5286c42108d88e7"
[[projects]]
name = "google.golang.org/grpc"
packages = [".","balancer","balancer/base","balancer/roundrobin","codes","connectivity","credentials","encoding","encoding/proto","grpclb/grpc_lb_v1/messages","grpclog","internal","keepalive","metadata","naming","peer","resolver","resolver/dns","resolver/passthrough","stats","status","tap","transport"]
revision = "8e4536a86ab602859c20df5ebfd0bd4228d08655"
version = "v1.10.0"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
inputs-digest = "1d4a82a70707701949a087ff1161b20ac92d1936d1571961cc82e3f5c54b5fe0"
solver-name = "gps-cdcl"
solver-version = 1
# Gopkg.toml example
#
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
#
# [prune]
# non-go = false
# go-tests = true
# unused-packages = true
[[constraint]]
name = "cloud.google.com/go"
version = "0.18.0"
[[constraint]]
name = "github.com/gorilla/mux"
version = "1.6.1"
[[constraint]]
name = "github.com/rs/cors"
version = "1.2.0"
[[constraint]]
name = "github.com/satori/go.uuid"
version = "1.2.0"
[[constraint]]
name = "github.com/stretchr/testify"
version = "1.2.0"
[[constraint]]
branch = "master"
name = "golang.org/x/net"
[[constraint]]
branch = "master"
name = "golang.org/x/oauth2"
[[constraint]]
branch = "master"
name = "google.golang.org/api"
[[constraint]]
name = "google.golang.org/appengine"
version = "1.0.0"
[prune]
go-tests = true
unused-packages = true
# DPS Entitlements
## API Authorization
Every API is authorized with the following steps:
1. data-partition-id
- Partition is passed from the applications making requests to the data ecosystem which ultimately is passed to entitlements.
2. JWT Bearer Token -- ESP validates the token
- The identity making the request to entitlement service.
3. Check if the user making the request belongs to entitlements.users@{x-account-id}.domain.com
- entitlements.users@{x-account-id}.domain.com is created by data manager and should already be available for the corresponding tenant.
4. Check if the user making the call belongs to users@{x-account-id}.domain.com
## Administration
Administration of entitlements is enforced by Google, implemented using Google Groups & GSuite Admin API SDK:
`https://developers.google.com/admin-sdk/`
*Refer to docs folder for flowcharts*
## Getting Started
### Google Cloud Setup
- Install the Google Cloud SDK from [https://cloud.google.com/sdk/downloads](https://cloud.google.com/sdk/downloads)
- Add the bin directory of the SDK to the `Path` environment variable, e.g. `C:\google-cloud-sdk\bin` if installed on your C:/ drive
- Run `gcloud init` to initialize the SDK
- Run `gcloud auth login` to log into your account on the SDK
- Run `gcloud auth application-default login` to acquire new user credentials to use for Application Default Credentials. This is required for your tests to run.
### Entitlements Setup
- Install `Go` language.
- Set `GOPATH` to the path to this project folder
- Set `GOROOT` to `C:\Go` (or wherever Go was installed on your machine)
- Set `GOBIN` to the value of your `GOPATH` with `/bin` appended to the end
- e.g. if `GOPATH` is `C:\workspace\dps-entitlements`, `GOBIN` would be `C:\workspace\dps-entitlements\bin`
- Install `dep`(Go dependency manager) from the root of this directory
- Copy `dep.exe` in the `bin` folder where Go language installed e.g. `C:\Go\bin`
- You will need following files from the google project :
- `src/entitlements/entitlements-bucket-access.json` can be found in bucket file `<project-name>-entitlements/entitlements-bucket-access.json`
- Change Redishost to `localhost` in `./src/entitlements/config/config.toml`
### Using Redis
- Install Redis
- Run Redis server on port 6379
### Running
- To run locally, it is required to set up Entitlements Cache Sync project. To setup the same please go through the readMe file.
- Install all dependencies `dep ensure` from `./src/entitlements`
- Use the `-v` flag for verbose output
- Run the app
- `cd src/entitlements`
- `go run app.go`
### Testing
- `go test ./...` runs all tests in the current directory
- use the `-v` flag for verbose output
- to force all tests to run, turn off the cache with `GOCACHE=off go test ./...`
### Integration Test
- Open directory `C:\workspace\dps-entitlements\dps-entitlements-integration-tests`
-`go test ./...` runs all tests in the current directory
- use the `-v` flag for verbose output
## Local Build Scripts
### Pre-reqs
- Install Docker
- Be a member of the GCP project
- Speak to someone on the team to add you to this project if you are not already.
### Steps
- From the directory that contains `docker-compose.yml` run `docker-compose up -d`
- If this is the first time running this:
- Run `gcloud docker --authorize-only` beforehand
- It will pull the image which will take several minutes (current size is 7.86GB)
- While the container is running, you can use the `docker exec` command to run scripts
- ex: `docker exec dpsentitlements_dev_1 build-scripts/build.sh`
- Can confirm the container is running with `docker ps`
- If this does not work:
- Make sure the .sh files in `build-scripts` have Unix line endings
- Run `mintty.exe -t "$PWD" winpty docker-compose exec dev bash`
## Logs-based Metrics
### Overview
- We are using gcloud logging to create logs-based metrics for tracking issues and firing alerts. Charts can be viewed from stackdriver.
- [introduction](https://cloud.google.com/logging/docs/logs-based-metrics/)
- [beta](https://cloud.google.com/sdk/gcloud/reference/beta/logging/metrics/)
- [stackdriver](https://app.google.stackdriver.com)
### How to manage
- We update our metrics for each environment as we deploy to them.
- To create a new metric, add a json file with the following structure to the metrics folder:
`{
"name": "example-name",
"description": "some description",
"filter": "gcp logs filter"
}`
- Name and filter must not be empty.
- Google cloud only supports updating description and filter. Changing the name of an existing metric will result in a new metric being created.
- New metrics only record new logs, they do not collect information that already exists.
- [Using Metric Explorer](https://cloud.google.com/logging/docs/logs-based-metrics/charts-and-alerts)
\ No newline at end of file
{{$endPointsServiceName := getenv "ENDPOINTS_SERVICE_NAME" -}}
service: entitlements
runtime: go
env: flex
endpoints_api_service:
name: "{{$endPointsServiceName}}"
rollout_strategy: managed
resources:
cpu: 1
memory_gb: 3.0
automatic_scaling:
min_num_instances: 4
max_num_instances: 30
cool_down_period_sec: 180
cpu_utilization:
target_utilization: 0.2
\ No newline at end of file
#!/bin/bash
# This script parses the report.xml generated via `go test` and fails if % of failing unit tests is above a threshold
report_file='../testresults/test.out'
fail_threshold_percentage=0
tests_passed=`cat $report_file | grep "PASS:" | wc -l`
tests_failed=`cat $report_file | grep "FAIL" | wc -l`
tests_total=`cat $report_file | grep "RUN" | wc -l`
percentage_failure=$tests_failed*100/$tests_total
percentage_failure=$(( $tests_failed * 100 / $tests_total ))
if [ $percentage_failure -gt $fail_threshold_percentage ]
then
echo "Total Test Cases: $tests_total"
echo "Passing Test Cases: $tests_passed"
echo "Failing Test Cases: $tests_failed"
echo "Percentage of tests failing: $percentage_failure"
echo "Too many test failures. Please fix the failing unit tests and proceed"
exit 1
else
echo "Build result: success"
fi
\ No newline at end of file
#!/bin/bash
# Exit as soon as command fails (add -v for debugging strict)
set -e
export GOPATH=$BUILD_REPOSITORY_LOCALPATH
export PATH=$PATH:$GOPATH:$GOPATH/bin
# Go to source directory
cd $BUILD_REPOSITORY_LOCALPATH
echo "Count lines of code"
mkdir -p cloc
cloc --vcs=git --md --out=${BUILD_REPOSITORY_LOCALPATH}/cloc/cloc.md .
# Upload Markdown
echo "##vso[task.addattachment type=Distributedtask.Core.Summary;name=Lines of Code;]${BUILD_REPOSITORY_LOCALPATH}/cloc/cloc.md"
# Run golint
golint -- || { echo '##vso[task.logissue type=warning]Lint failed'; }
echo 'installing go dep'
go get -u github.com/golang/dep/cmd/dep
echo 'go dep installed'
# Run dep ensure to install dependencies (this step creates a 'vendor' folder)
echo 'running dep ensure'
pushd src/entitlements
rm -rf vendor
dep ensure -v
echo 'dep ensure complete'
popd
# Create testresult and cobertura folders
mkdir -p testresults
mkdir -p cobertura
# Testing
echo "Testing"
. ${BUILD_REPOSITORY_LOCALPATH}/build-scripts/cover.sh . ./src/entitlements/ --html
# Copy artifacts
echo "Copying artifacts"
find .cover/*.xml -size 65c -delete #delete empty test results (no tests run)
cp .cover/*.xml testresults
cp .cover/*.html testresults
gocov convert .cover/cover.out | gocov-xml > cobertura/coverage.xml
# Transform the coverage report to VSTS compatible DTD format
echo "Transforming the coverage report to VSTS compatible DTD format"
python ./build-scripts/forcobertura.py cobertura/coverage.xml
rm cobertura/coverage.original.xml
# Run the build.
# Run the build. This creates the bin and pkg directories
pushd src
go install entitlements
popd
\ No newline at end of file
#!/bin/bash
# This should never be empty, unless you are in a local environment. $GCLOUD_PROJECT is injected as env var during release
if [ "$GCLOUD_PROJECT" = "" ]
then
export GCLOUD_PROJECT=$(gcloud config get-value project)
fi
\ No newline at end of file
#!/bin/sh
# Generate test coverage statistics for Go packages.
#
# Usage: cover go_path app_path [--html | --coveralls]
#
# app_path Path to the root folder for test discovery
# --html Additionally create HTML report and open it in browser
# --coveralls Push coverage statistics to coveralls.io
#
set -e
# GOPATH
go_path=`cd "${1}"; pwd` # convert relative path to absolute
export GOPATH=${go_path}
app_path=${2}
format=${3}
work_dir=.cover
profile="$work_dir/cover.out"
html="$work_dir/index.html"
mode=count
generate_coverage_data() {
rm -rf "$work_dir"
mkdir "$work_dir"
status=0
outFile="$work_dir/entitlements.out"
go clean -cache
go test "$app_path/..." -v -coverprofile="$profile" > ${outFile} || true
cat ${outFile} >> test.out
cat ${outFile} | go-junit-report > "$work_dir/entitlements.xml"; status=$(($status|$?))
rm ${outFile}
# move test.out to testresults
mv test.out testresults/test.out
if [ ${status} -ne 0 ]
then exit ${status}
fi
}
show_cover_report() {
go tool cover -$1="$profile" -o ${html}
}
push_to_coveralls() {
echo "Pushing coverage statistics to coveralls.io"
goveralls -coverprofile="$profile"
}
generate_coverage_data
show_cover_report func
case "$format" in
"")
;;
--html)
show_cover_report html ;;
--coveralls)
push_to_coveralls ;;
*)
echo >&2 "error: invalid option: $format"; exit 1 ;;
esac
#!/bin/bash
# deploy script
# Exit as soon as a command fails
set -e
deploy_endpoint() {
sed -i -e "s/YOUR-PROJECT-ID/$GCLOUD_PROJECT/g" entitlements.yaml
sed -i -e "s/YOUR-AUDIENCES/$GOOGLE_AUDIENCES/g" entitlements.yaml
echo "Print Swagger Json"
cat entitlements.yaml
echo "Deploying Endpoints"
gcloud endpoints services deploy entitlements.yaml --project=$GCLOUD_PROJECT
rm entitlements.yaml
}
deploy_endpoint
\ No newline at end of file
#!/bin/bash
# Exit as soon as a command fails
set -e
chmod a+x build-scripts/*.sh
source build-scripts/config.sh
echo "Deploying to gcloud"
build-scripts/deploy2gcp.sh "$@"
echo "Deployed to gcloud"
echo "Setting logs-based metrics"
build-scripts/metrics.sh
echo "Finished setting logs-based metrics"
\ No newline at end of file