使用作业书签的AWS Glue失败，并显示“数据源不支持写入空或嵌套的空模式”

debugcn 发表于 Dev

栾凯文·费雷拉

我正在使用书签功能进行胶水（2.0版）作业，第二次运行该作业而文件没有任何更改时，出现以下错误消息：

AnalysisException: '\nDatasource does not support writing empty or nested empty schemas.\nPlease make sure the data schema has at least one or more column(s).\n ;'

它是AWS控制台生成的脚本，未经任何修改，源是使用数据目录的S3文件，输出是另一个存储桶。

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "raw_dev_edocs", table_name = "esocial_s2200", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "raw_dev_edocs", table_name = "esocial_s2200", transformation_ctx = "datasource0")
## @type: ApplyMapping
## @args: [mapping = [("esocial", "struct", "esocial", "struct"), ("tenant", "string", "tenant", "string"), ("year", "string", "year", "string"), ("month", "string", "month", "string"), ("day", "string", "day", "string")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("esocial", "struct", "esocial", "struct"), ("tenant", "string", "tenant", "string"), ("year", "string", "year", "string"), ("month", "string", "month", "string"), ("day", "string", "day", "string")], transformation_ctx = "applymapping1")
## @type: ResolveChoice
## @args: [choice = "make_struct", transformation_ctx = "resolvechoice2"]
## @return: resolvechoice2
## @inputs: [frame = applymapping1]
resolvechoice2 = ResolveChoice.apply(frame = applymapping1, choice = "make_struct", transformation_ctx = "resolvechoice2")
## @type: DropNullFields
## @args: [transformation_ctx = "dropnullfields3"]
## @return: dropnullfields3
## @inputs: [frame = resolvechoice2]
dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")
## @type: DataSink
## @args: [connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4"]
## @return: datasink4
## @inputs: [frame = dropnullfields3]
datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4")
job.commit()

如本文档（https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html）所述，书签使用修改时间戳来跟踪，我的脚本与文档示例相同。

当我重置书签时，作业成功运行。

Glue从中读取的表模式是：

Column name
Data type
Key
Comment
1
tenant
string
Partition (0)

2
year
string
Partition (1)

3
month
string
Partition (2)

4
day
string
Partition (3)

5
esocial.xmlns
string

6
esocial.evtadmissao.id
string

7
esocial.evtadmissao.ideevento.indretif
string

8
esocial.evtadmissao.ideevento.tpamb
string

9
esocial.evtadmissao.ideevento.procemi
string

10
esocial.evtadmissao.ideevento.verproc
string

11
esocial.evtadmissao.ideevento.nrrecibo
string

12
esocial.evtadmissao.ideempregador.tpinsc
string

13
esocial.evtadmissao.ideempregador.nrinsc
string

14
esocial.evtadmissao.trabalhador.cpftrab
string

15
esocial.evtadmissao.trabalhador.nistrab
string

16
esocial.evtadmissao.trabalhador.nmtrab
string

17
esocial.evtadmissao.trabalhador.sexo
string

18
esocial.evtadmissao.trabalhador.racacor
string

19
esocial.evtadmissao.trabalhador.estciv
string

20
esocial.evtadmissao.trabalhador.grauinstr
string

21
esocial.evtadmissao.trabalhador.nascimento.dtnascto
string

22
esocial.evtadmissao.trabalhador.nascimento.codmunic
string

23
esocial.evtadmissao.trabalhador.nascimento.uf
string

24
esocial.evtadmissao.trabalhador.nascimento.paisnascto
string

25
esocial.evtadmissao.trabalhador.nascimento.paisnac
string

26
esocial.evtadmissao.trabalhador.nascimento.nmmae
string

27
esocial.evtadmissao.trabalhador.nascimento.nmpai
string

28
esocial.evtadmissao.trabalhador.documentos.ctps.nrctps
string

29
esocial.evtadmissao.trabalhador.documentos.ctps.seriectps
string

30
esocial.evtadmissao.trabalhador.documentos.ctps.ufctps
string

31
esocial.evtadmissao.trabalhador.documentos.rg.nrrg
string

32
esocial.evtadmissao.trabalhador.documentos.rg.orgaoemissor
string

33
esocial.evtadmissao.trabalhador.documentos.rg.dtexped
string

34
esocial.evtadmissao.trabalhador.documentos.rne.nrrne
string

35
esocial.evtadmissao.trabalhador.documentos.rne.orgaoemissor
string

36
esocial.evtadmissao.trabalhador.documentos.rne.dtexped
string

37
esocial.evtadmissao.trabalhador.endereco.brasil.tplograd
string

38
esocial.evtadmissao.trabalhador.endereco.brasil.dsclograd
string

39
esocial.evtadmissao.trabalhador.endereco.brasil.nrlograd
string

40
esocial.evtadmissao.trabalhador.endereco.brasil.bairro
string

41
esocial.evtadmissao.trabalhador.endereco.brasil.cep
string

42
esocial.evtadmissao.trabalhador.endereco.brasil.codmunic
string

43
esocial.evtadmissao.trabalhador.endereco.brasil.uf
string

44
esocial.evtadmissao.trabalhador.endereco.brasil.complemento
string

45
esocial.evtadmissao.trabalhador.infodeficiencia.deffisica
string

46
esocial.evtadmissao.trabalhador.infodeficiencia.defvisual
string

47
esocial.evtadmissao.trabalhador.infodeficiencia.defauditiva
string

48
esocial.evtadmissao.trabalhador.infodeficiencia.defmental
string

49
esocial.evtadmissao.trabalhador.infodeficiencia.defintelectual
string

50
esocial.evtadmissao.trabalhador.infodeficiencia.reabreadap
string

51
esocial.evtadmissao.trabalhador.infodeficiencia.infocota
string

52
esocial.evtadmissao.trabalhador.contato.foneprinc
string

53
esocial.evtadmissao.trabalhador.contato.fonealternat
string

54
esocial.evtadmissao.trabalhador.contato.emailprinc
string

55
esocial.evtadmissao.trabalhador.contato.emailalternat
string

56
esocial.evtadmissao.trabalhador.dependente.struct.tpdep
string

57
esocial.evtadmissao.trabalhador.dependente.struct.nmdep
string

58
esocial.evtadmissao.trabalhador.dependente.struct.dtnascto
string

59
esocial.evtadmissao.trabalhador.dependente.struct.cpfdep
string

60
esocial.evtadmissao.trabalhador.dependente.struct.depirrf
string

61
esocial.evtadmissao.trabalhador.dependente.struct.depsf
string

62
esocial.evtadmissao.trabalhador.dependente.struct.inctrab
string

63
esocial.evtadmissao.trabalhador.dependente.array
bigint

64
esocial.evtadmissao.trabalhador.indpriempr
string

65
esocial.evtadmissao.trabalhador.trabestrangeiro.dtchegada
string

66
esocial.evtadmissao.trabalhador.trabestrangeiro.classtrabestrang
string

67
esocial.evtadmissao.trabalhador.trabestrangeiro.casadobr
string

68
esocial.evtadmissao.trabalhador.trabestrangeiro.filhosbr
string

69
esocial.evtadmissao.vinculo.matricula
string

70
esocial.evtadmissao.vinculo.tpregtrab
string

71
esocial.evtadmissao.vinculo.tpregprev
string

72
esocial.evtadmissao.vinculo.cadini
string

73
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.dtadm
string

74
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.tpadmissao
string

75
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.indadmissao
string

76
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.tpregjor
string

77
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.natatividade
string

78
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.dtbase
string

79
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.cnpjsindcategprof
string

80
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.fgts.opcfgts
string

81
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.fgts.dtopcfgts
string

82
esocial.evtadmissao.vinculo.infocontrato.codcargo
string

83
esocial.evtadmissao.vinculo.infocontrato.codcateg
string

84
esocial.evtadmissao.vinculo.infocontrato.remuneracao.vrsalfx
string

85
esocial.evtadmissao.vinculo.infocontrato.remuneracao.undsalfixo
string

86
esocial.evtadmissao.vinculo.infocontrato.duracao.tpcontr
string

87
esocial.evtadmissao.vinculo.infocontrato.duracao.dtterm
string

88
esocial.evtadmissao.vinculo.infocontrato.duracao.clauassec
string

89
esocial.evtadmissao.vinculo.infocontrato.localtrabalho.localtrabgeral.tpinsc
string

90
esocial.evtadmissao.vinculo.infocontrato.localtrabalho.localtrabgeral.nrinsc
string

91
esocial.evtadmissao.vinculo.infocontrato.horcontratual.qtdhrssem
string

92
esocial.evtadmissao.vinculo.infocontrato.horcontratual.tpjornada
string

93
esocial.evtadmissao.vinculo.infocontrato.horcontratual.tmpparc
string

94
esocial.evtadmissao.vinculo.infocontrato.horcontratual.horario
bigint

95
esocial.evtadmissao.vinculo.infocontrato.filiacaosindical.cnpjsindtrab
string

96
esocial.evtadmissao.vinculo.infocontrato.observacoes.observacao
string

97
esocial.evtadmissao.vinculo.afastamento.dtiniafast
string

98
esocial.evtadmissao.vinculo.afastamento.codmotafast
string

有谁知道我该如何解决这个问题？

栾凯文·费雷拉

我做了一个解决方法来解决问题。

if len(dropnullfields3.select('root').toDF().schema) > 0:
    datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4")
    job.commit()

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。